In late September 2026 I wrote a LinkedIn note dissecting Mixture‑of‑Experts (MoE) routing, highlighting that each token picks two experts, the GPU multiplies the selected expert outputs by their routing weights, adds them, and then restores the token order. Because different tokens can choose different expert pairs, a single batch may involve every GPU storing experts, leading to all‑to‑all activation exchange across GPUs.
GPU 0 multiplies both selected expert outputs by their routing weights, adds them together, and restores the original token order.
Different tokens can select different expert pairs.
Across a batch, those assignments may involve every GPU storing experts.
Each GPU may therefore send activations to several GPUs while receiving activations for its own experts.
This exchange is called all-to-all communication.
Keeping frequently selected experts within the same server reduces remote traffic.
Balanced routing prevents one GPU from delaying the layer.
Communication overlap allows local expert computation to continue while remote activations move.
Sparse routing reduces expert FLOPs, but token dispatch and network communication erase part of that saving.
I linked this note from my broader September 2026 LinkedIn recap page for context: September 2026 posts summary.