/learn

Inference, chapter by chapter

Read in order: each chapter builds on the one before it. Toggle layers (Concept / Maths / Code) inside any chapter to choose how deep to go. New to transformers? Start with the Transformer Decoder Explainer, which walks through one forward pass.

1. 01From forward pass to generation loop

   Prefill, decode, sampling and stopping: why generation is sequential.

2. 02The KV cache

   What is cached, why it is exact, and how big it gets.

3. 03Arithmetic intensity and the roofline

   Why decode is memory-bound and prefill compute-bound.

4. 04Batching

   Static, dynamic, continuous and chunked prefill.

5. 05Memory management

   PagedAttention, block tables, prefix sharing and preemption.

6. 06Attention kernels

   FlashAttention's IO-aware tiling; split-K for long contexts.

7. 07Speculative decoding

   Draft, verify, and the expected tokens per step.

8. 08Quantisation for inference

   Fewer bytes per weight and per cached value, faster decode.

9. 09Parallelism

   Tensor, pipeline and expert parallelism and what they send.

10. 10Serving metrics

    TTFT, TPOT, ITL, goodput, tail latency and Little's law.

11. 11Why disaggregate

    Prefill stalls decode on a shared GPU; separate pools stop it.

12. 12Moving the KV cache

    Hand-off size, link bandwidth, streaming, compression, mixed pools.

13. 13A live disaggregated simulator

    Pools, devices, links, load and SLOs: run the simulator yourself.

14. 14Trade-offs: when not to disaggregate

    Small models, low load and slow links, and where to go next.