Choosing inference techniques by bottleneck

We met on 7 September 2026 to discuss chapter 5 of Inference Engineering by Philip Kiely. We discussed four ways to reduce inference cost or hardware pressure. Quantization, caching, speculative decoding, and model parallelism with disaggregation each address a different constraint.

Lessons from the session

The discussion left us with these lessons.

• The four techniques address different constraints. We can shrink the model with quantization, reuse work with caching, generate candidate tokens using a draft model, or distribute computation across devices and separate prefill from decode.
• Speculative decoding uses a fast draft model to propose tokens. A larger target model verifies those tokens and accepts the prefix it would also have generated. A smaller model from the same family is a common draft-model choice.
• Quantization is more than changing 32-bit weights to 16-bit weights. The actual memory and speed change depends on the method and on which layers are quantized, and output quality can change as well.
• The techniques share one constraint: the model's requirements may exceed the GPU memory or compute available. The right choice depends on the bottleneck and the workload.

Next actions

Use this checklist before the modalities discussion.

• [ ] Read chapter 6 on modalities.
• [ ] Write down your bottleneck before choosing a technique. Name whether you need to save model memory, reuse prompt work, speed up token generation, or split work across devices.
• [ ] For a quantization experiment, use the same model and fixed prompts at two precisions. Record VRAM use, latency, output tokens per second, and a small quality check.
• [ ] For speculative decoding, draw the draft-model and target-model sequence. Mark which proposed tokens the target accepts and which it regenerates.
• [ ] Add one question to the book page, especially if a technique still feels abstract.

Recording

Watch the Week 4 recording on YouTube.

Recording chapters

Use these timestamps to jump to the main parts of the discussion.

• 0:07 - The four techniques in the chapter
• 4:21 - Draft and target models in speculative decoding
• 12:38 - Why quantization is more nuanced than fewer bits
• 19:22 - Finding a practical way to experiment
• 20:29 - Preparing for modalities
• 21:54 - The final two chapters

Continue with Week 5

The next discussion covered chapter 6 on modalities. Read the Week 5 recap or browse the book-club series.