Parallelism

Llama-3-70B's weights alone are 141.10 GB in BF16: they don't fit on one 80 GB GPU, before any KV cache. Even when a model fits, spreading it over several GPUs adds their memory bandwidth together, and decode is bandwidth-bound. There are three main ways to split a model, and each one adds a different kind of message.

Tensor parallelism (TP) splits every weight matrix across the $n$ devices. Megatron-LM's arrangement splits the attention heads and the MLP's hidden dimension so that each block needs just two all-reduces in the forward pass: one after attention, one after the MLP. Every device holds $1/n$ of every layer and does $1/n$ of every multiply. The messages are small (one activation vector per token) but there are many of them, twice per layer, on the critical path of every step. TP wants the fastest link you have, which is why it normally stays inside one NVLink-connected server.

Pipeline parallelism (PP) gives each device a contiguous slice of the layers. A token's activations pass from stage to stage: one message per boundary per step. Cheap on the link, so PP can span slower links between servers. The catch is the bubble: stage 2 can't start until stage 1 finishes, so with a single batch only one stage is busy at a time. Keeping several micro-batches in flight fills the pipeline; GPipe's bubble fraction is $(n-1)/(m+n-1)$ for $n$ stages and $m$ micro-batches.

Expert parallelism (EP) applies to mixture-of-experts models, whose MLP is many "expert" MLPs and a router that sends each token to its top-$k$. Different devices hold different experts, so every MoE layer needs an all-to-all: send each token to the devices holding its experts (dispatch), and bring the results back (combine).

Real deployments mix them, for example TP inside a server, PP or EP across servers, and data parallelism (independent replicas) on top.

With the simulator's link figures and this page's first-order model, a Llama-3-70B decode step at batch 16 on 4× H100 takes 14.47 ms of compute, and TP's 160 all-reduces add 4.94 ms over NVLink but 24.23 ms over 100 GbE. Notice that on NVLink almost all of it (4.80 ms) is the per-message latency term, not bandwidth: decode messages are tiny. That is a pessimistic estimate (collective libraries have low-latency algorithms for small messages, and engines overlap communication with compute), but the direction is right: decode-time TP is latency-sensitive, and slow links make it impractical. A pipeline boundary over InfiniBand, by contrast, moves 256 KiB per step in about 46 µs.