LLM Inference Engineer

Prepare for the interview that asks how many users your GPUs can serve.

LLM inference interviews are unlike other engineering interviews. You may be asked how much memory a model needs to serve a hundred users at 8,000 tokens of context, why time to first token tripled when a customer started sending long documents, or to write the part of a serving engine that decides which requests run next and what happens when memory runs out. Each question has an answer you can reach with a minute of arithmetic and a precise understanding of a few mechanisms.

This book teaches both. It builds a working miniature inference engine, one mechanism per chapter, and derives every capacity, latency and cost figure from first principles, so you can reason about any model on any hardware.

Inside you'll find:

• The arithmetic of inference: prefill and decode, hardware bounds, TTFT, TPOT, goodput and Little's law, with worked estimation drills
• The engine, in working code: the KV cache, paged memory with prefix caching and copy-on-write, continuous batching with chunked prefill and preemption, sampling and structured output, speculative decoding and quantisation
• Scaling out: tensor, pipeline and expert parallelism, mixture-of-experts models, disaggregated prefill and decode, and long context
• The serving system: routing, rate limiting, autoscaling, multi-LoRA serving, capacity planning, cost per token, benchmarking and observability
• Interview practice: the coding round with tested solutions, 20 debugging scenarios, 8 design questions, rubrics of strong signals and red flags, and over 220 questions with answers
• Reference appendices: GPU and model tables, engine settings, a glossary and further reading

Code you can trust. Every listing comes from a small engine that runs on an ordinary laptop, and its tests check that batching, chunking, preemption and prefix caching never change a single output token compared with the simplest possible generation loop. Performance figures are bounds computed from published specifications, with the arithmetic shown, never invented benchmarks.

Who it's for: engineers preparing for LLM inference, model serving and inference-optimisation roles, including backend and infrastructure engineers moving into AI, ML engineers moving towards serving, and GPU engineers moving up the stack. You should be able to read Python. Book Three of The AI Infrastructure Library; it can be read on its own.