Inference Engineering Book Club - Week 3

We met on 1 September 2026 to discuss chapter 4 of Inference Engineering by Philip Kiely. We used the chapter to compare serving tools. Before choosing one, define the workload and constraints, benchmark candidates, and record the trade-off.

Lessons from the session

The discussion left us with these lessons.

We compared vLLM, SGLang, TensorRT-LLM, and Llama.cpp. No tool wins for every model and workload. The choice also depends on hardware, latency, throughput, license, operational constraints, and team experience.
Theoretical estimates give us a starting point, but they don't establish the operating limits of a serving system. One participant compared estimates with measurements from a rented GPU and found that the real capacity differed from the formula-based estimate.
Shadow traffic provides a safer comparison for a live service. We send a copy of real requests to a candidate while keeping the current provider on the user-facing path, then compare latency, throughput, and behavior.
vllm bench can help measure prefill and decode behavior. Setting a target such as time to first token gives the benchmark a decision to answer instead of producing numbers without context.

Next actions

Use this checklist before the techniques discussion.

[ ] Read chapter 5 on inference techniques.
[ ] Pick a baseline serving setup and one candidate. Write down the model, hardware, precision, prompt and output lengths, concurrency, and workload sample before measuring.
[ ] Choose the metrics that decide the choice: time to first token, output tokens per second, total throughput, error rate, and output quality where relevant.
[ ] Run an offline benchmark first. If you have a live service and a safe environment, use shadow traffic instead of routing users to an untested candidate.
[ ] Record the target and the measured result together. Use the gap to decide what to change next, such as batching, concurrency, or another serving tool.
[ ] Add your benchmark assumptions to the book page so another reader can reproduce the comparison.

Resources

These links provide the reading context for the session.

Inference Engineering.
AI Shipping Labs book page for chapter notes and the shared reading progress.

Recording

Watch the Week 3 recording on YouTube.

Recording chapters

Use these timestamps to jump to the main parts of the discussion.

Continue with Week 4

The next discussion covered chapter 5 on techniques. Read the Week 4 recap or browse the book-club series.