In October 2026 I published a LinkedIn note summarising the six main parallelism strategies for large‑language‑model inference and how I decide which to apply in production.
Putting a model on four GPUs does not create one larger GPU.
Data parallelism runs complete model replicas on separate GPU groups. It increases throughput, but cannot make an oversized model fit.
Tensor parallelism splits matrix operations inside each transformer layer. It reduces weight memory per GPU, but the GPUs must combine partial results throughout the layer stack.
Pipeline parallelism assigns layer groups to different GPUs. Activations move between stages, while insufficient micro‑batching leaves some stages idle.
Prefill context parallelism splits a long prompt across GPUs. It targets time to first token when attention over the input dominates latency.
Decode context parallelism splits the stored KV cache across GPUs.
Expert parallelism distributes MoE experts across GPUs. Token routing and uneven expert load add network cost and idle time.
Keep tensor parallelism on the fastest links because it communicates inside many layers.
I usually combine several strategies, starting with tensor parallelism on the fastest interconnects, then adding data parallelism only after a single replica fits, and scaling further only when measured communication cost is lower than the resource limitation being removed.