Inference Engineering Book Club - Week 1

We met on 17 August 2026 after reading chapters 0, 1, and 2 of Inference Engineering by Philip Kiely. We connected the book's model and hardware concepts to decisions about model inputs, serving, and measurement.

Use the AI Shipping Labs book page to mark the chapters you have read and add a question or explanation for the next reader.

Lessons from the session

The discussion left us with these lessons.

Chapters 0 and 1 frame inference as a series of product and engineering decisions. Start with the use case, budget, model choice, and evaluation before optimizing.
Chapter 2 introduces attention, logits, arithmetic intensity, and the KV cache. Several readers found the transition from the high-level chapters to model internals abrupt, so it's reasonable to keep moving and revisit the hard sections when later chapters need them.
The roofline model helps explain where inference time goes. Prefill processes the prompt in a larger batch and tends to be compute-bound. Decode generates one token at a time and repeatedly reads model state, so memory movement becomes the constraint.
A long-document tender workflow showed why sending every page to a model may be the wrong first step. Before automating, write down how a person reads the documents, what they skip, and what information they extract before deciding.

Next actions

Use this checklist before the hardware discussion.

[ ] Read chapter 3 on hardware.
[ ] Draw a small prefill-versus-decode diagram. Mark where the model reads the prompt, generates tokens, and reuses the KV cache.
[ ] For one long-document task, write the current human workflow in five steps or fewer. Record what the person reads first, what they extract, and what decision they make.
[ ] Identify one place where a summary, retrieval step, or focused extraction could reduce the context sent to the final model. Treat that as a hypothesis to test, not an assumption.
[ ] Add your hardest question to the chapter notes so the next discussion can work from concrete examples.

Resources

These links provide follow-up reading for the discussion.

Inference Engineering.
Anthropic's multi-agent research system, referenced when discussing focused extraction from large documents.
Claude text watermarking by Sebastian Raschka, a useful connection to sampling concepts from chapter 2.
The official Claude text watermarking announcement.

Recording

Watch the Week 1 recording on YouTube.

Recording chapters

Use these timestamps to jump to the main parts of the discussion.

Continue with Week 2

The next discussion covered chapter 3 on hardware. Read the Week 2 recap or follow the book-club series.