Inference Engineering Book Club - Week 2

We met on 25 August 2026 to discuss chapter 3 of Inference Engineering by Philip Kiely. We discussed GPUs, nodes, interconnects, and the memory limits that influence model-serving decisions.

Before you load a model, estimate whether it and its workload will fit on the hardware.

Lessons from the session

The discussion left us with these lessons.

GPUs handle the parallel matrix and vector operations that dominate many model workloads, while CPUs are better suited to more general-purpose work.
A serving system may use several GPUs in one node and several nodes connected together. The links between those devices matter because the model and its intermediate data must move between them.
Model parameters don't tell you the full memory requirement. We also need room for the KV cache and the serving runtime.
Trial and error isn't a planning method. Loading a model, hitting an out-of-memory error, and trying a more heavily quantized version tells us what failed, not what will fit. Make an estimate first, then measure the actual workload.

Next actions

Use this checklist before the software discussion.

[ ] Read chapter 4 on software.
[ ] Pick one model and one target GPU. Write down the parameter count, weight precision, expected KV-cache size, runtime overhead, and available VRAM. Label each number as an estimate or a measurement.
[ ] Describe the workload that the GPU must serve: prompt length, output length, concurrency, and the latency or throughput target.
[ ] Compare your estimate with an actual load or benchmark when you can. Record what the estimate missed instead of treating the first out-of-memory error as the sizing method.
[ ] Add one hardware question to the book page for the software discussion.

Recording

Watch the Week 2 recording on YouTube.

Recording chapters

Use these timestamps to jump to the main parts of the discussion.

Continue with Week 3

The next discussion covered chapter 4 on serving software. Read the Week 3 recap or browse the book-club series.