Expert-Parallel MoE Inference: Serving Sparse Models at Scale

The models topping the 2026 open-weight leaderboards — GLM, Kimi, DeepSeek, Qwen and their Mixtral-class ancestors — share one architectural fact that quietly reshapes every serving decision you make: they are sparse Mixture-of-Experts networks with hundreds of billions of parameters but only a small slice active per token. That gap is exactly what makes expert-parallel MoE inference both attractive and treacherous. You get the quality of a huge model at the compute cost of a small one, but only if your serving stack can move tokens to the right experts, across the right GPUs, without drowning in communication. Get the topology wrong and a 30B-active model runs slower than a 70B dense one on the same hardware.

This is a serving problem, not a model problem. The weights are fixed; what you control is how experts are sharded, how tokens are routed between them, and how you overlap that routing with useful math.

What this covers: why MoE decouples parameters from FLOPs, how expert parallelism differs from tensor and pipeline parallelism, why the all-to-all dispatch and combine dominate latency, how load imbalance and token dropping bite in production, and the concrete trade-offs behind prefill/decode disaggregation, expert offload, and EPLB-style balancing.

Context and Background

A dense transformer runs every parameter for every token. A Mixture-of-Experts model replaces the feed-forward block in each layer with many parallel FFN “experts” and a small router (gate) that picks the top-k experts per token — typically k of 8 out of 128 or 256. The attention layers stay dense; only the FFN becomes sparse. The result is a model whose total parameter count can be 10-20x its active parameter count. DeepSeek-V3, for example, carries 671B total parameters but activates only ~37B per token, and the same decoupling logic drives the GLM, Kimi and Qwen MoE families that dominate 2026 open-weight releases.

This decoupling is the whole point. Quality scales with total parameters (knowledge capacity); compute cost scales with active parameters (FLOPs per token). Training papers from GShard and the Switch Transformer onward showed you can grow one without the other. But serving inherits a nasty side effect: those hundreds of billions of parameters still have to live in GPU memory somewhere, even though each token touches only a few of them. You cannot fit 671B parameters in weights on one 80GB GPU. So the experts get spread across many GPUs, and every token now has to travel to wherever its chosen experts physically sit. That routing traffic — not the matrix multiplies — is what your serving architecture has to master. It is a different discipline from dense serving, and it interacts tightly with everything from batching to prefill/decode disaggregation.

The state of the art in 2026 has converged on a handful of patterns that this article unpacks: sharding experts across a high-bandwidth GPU domain, dedicated all-to-all libraries that overlap communication with compute, runtime load balancers that reshuffle experts to flatten skew, and disaggregated serving that lets prefill and decode each pick their own parallel layout. What used to be bespoke infrastructure inside a few frontier labs is now largely reproducible with open frameworks — vLLM, SGLang and TensorRT-LLM all ship first-class expert-parallel modes — plus open communication kernels like DeepEP and open placement tools like EPLB. The hard part is no longer whether these pieces exist; it is understanding how they interact so you can size a deployment that hits your latency and cost targets instead of one that thrashes on all-to-all.

The Expert-Parallel Reference Architecture

Expert parallelism (EP) shards the experts of each MoE layer across GPUs: GPU 0 holds experts 0-31, GPU 1 holds 32-63, and so on. Attention and the router are replicated on every GPU. When a token needs experts that live on other GPUs, the system performs an all-to-all exchange to send each token’s hidden state to the GPU that owns its target expert, computes the FFN there, then does a second all-to-all to bring the results home. That dispatch-compute-combine loop is the beating heart of MoE serving.

Figure 1: The expert-parallel reference topology. Attention and the router run on every GPU; experts are partitioned across GPUs; two all-to-all collectives bracket the expert compute.

Long description: tokens enter a replicated attention layer, pass through the router gate which selects top-k experts, then an all-to-all dispatch fans each token out to the GPU owning its expert. Four GPUs each hold a disjoint slice of experts, run the FFN locally, and a second all-to-all combines outputs before the next attention layer.

Why expert parallelism and not just tensor parallelism

Dense models are usually sharded with tensor parallelism (TP), which splits every weight matrix across GPUs so each GPU does a fraction of every matmul, then all-reduces the partial sums. TP works beautifully for attention and for a single dense FFN, but it is wasteful for MoE. If you tensor-parallelise 256 experts across 8 GPUs, every GPU holds a shard of every expert, and you pay all-reduce traffic proportional to the full FFN even though each token only used k of them. Expert parallelism instead gives each GPU whole experts, so a GPU only does work for tokens actually routed to its experts. The communication becomes an all-to-all sized to the routed tokens, not an all-reduce sized to the full hidden dimension.

In practice you compose them. A large MoE deployment might run tensor parallelism inside the attention layers and across large shared/dense experts, expert parallelism across the routed experts, and pipeline parallelism across layers when the model is too deep for one node. The EP degree is often set equal to, or a multiple of, the number of experts divided by GPUs so that each GPU owns a clean integer number of experts. Frameworks such as vLLM, SGLang and TensorRT-LLM all expose an explicit expert-parallel size distinct from their TP and PP degrees precisely because these axes have different communication signatures.

The two collectives that define your latency

Every MoE layer issues two all-to-all collectives: a dispatch (scatter each token to its expert’s GPU) and a combine (gather the expert outputs back to the token’s home GPU). Unlike an all-reduce, an all-to-all’s cost depends on how tokens distribute across destinations, which the router decides at runtime and which you cannot know ahead of time. On a single well-connected node with NVLink or NVSwitch this is fast. The moment expert parallelism spans nodes over InfiniBand or Ethernet, all-to-all becomes the dominant cost — frequently larger than the expert matmul itself for the small batch sizes typical of decode. This is why the DeepSeek team built and open-sourced DeepEP, a communication library dedicated to making MoE dispatch and combine fast, with separate kernels tuned for high-throughput prefill and low-latency decode.

Where the parameters live

A useful mental model: attention weights and the KV cache are per-GPU replicated or TP-sharded and scale with sequence length and batch; expert weights are EP-sharded and scale with total parameter count. On a 671B-class model served across 16 or 32 GPUs, the bulk of HBM goes to expert weights sitting idle most of the time, waiting for the few tokens that route to them. That idle-but-resident property is what makes MoE serving a memory-capacity game as much as a compute game, and it is the reason single-GPU or small-cluster serving of large MoE models often turns to expert offloading, covered below.

Deeper Analysis: Routing, Balance, and a Capacity Worked Example

Under the hood, each MoE layer runs a tiny linear gate that scores every expert for every token, applies a softmax, and keeps the top-k. Those k experts each transform the token, and their outputs are combined weighted by the gate scores. The routing is data-dependent and, crucially, not uniform — some experts are far more popular than others, and that imbalance is the single biggest operational headache in expert-parallel serving.

Figure 2: The per-token routing path. The gate scores experts, top-k are chosen, a capacity check decides whether the token is served or dropped/rerouted, and outputs are combined by gate weight.

Long description: a token hidden state enters a gate linear producing softmax scores, top-k experts are selected, each routed token checks whether the target expert still has a free capacity slot, tokens that fit are dispatched to the expert FFN while overflow tokens are dropped or rerouted, and all results are combined weighted by gate score.

Capacity factor and token dropping

To keep tensor shapes static and memory bounded, MoE kernels assign each expert a fixed capacity — the maximum number of tokens it will process this batch, computed as capacity = capacity_factor * (num_tokens * k / num_experts). A capacity factor of 1.0 means “provision exactly the average load.” But routing is skewed, so a hot expert will exceed its slots while cold experts sit empty. Overflow tokens are dropped (they skip the FFN and pass through via the residual connection) or rerouted to their next-best expert. Dropping is cheap and keeps shapes fixed, but it silently degrades quality. In inference, most production stacks prefer a drop-free or dynamic-capacity path that pads to the actual max load, trading a little wasted compute for correctness. The knob matters: a capacity factor of 1.0 might drop several percent of tokens on a skewed batch, while 1.25-2.0 sharply cuts drops at the cost of more padding and communication.

The all-to-all dispatch and combine, step by step

Figure 3 shows the collective that dominates cross-node MoE latency. Each GPU simultaneously sends its locally-produced tokens to whichever GPUs own the destination experts, and receives tokens whose experts it owns. After local FFN compute, the reverse exchange returns results.

Figure 3: All-to-all dispatch and combine across three GPUs. Every GPU both sends and receives in each phase; expert FFN runs locally between the two collectives.

Long description: GPU0 dispatches tokens to GPU1 and GPU2 for experts hosted there while GPU1 and GPU2 dispatch tokens back to GPU0, each GPU then runs its expert FFN locally, and a second all-to-all combine returns every expert output to the GPU that originally held the token.

The killer property is that all-to-all is a synchronising collective: the slowest GPU-to-GPU transfer gates the whole layer, and there are two of them per MoE layer, per forward pass. A 60-layer MoE model does 120 all-to-all collectives to generate a single token during decode. This is why overlap matters so much. DeepEP-style libraries overlap the dispatch/combine communication with the expert compute and with attention of adjacent micro-batches, so the network transfer hides behind math instead of stalling in front of it. Without overlap, decode latency on a multi-node MoE deployment can be dominated by network round-trips rather than FLOPs.

Load imbalance and how to fight it

Because a handful of experts attract a disproportionate share of tokens, the GPUs hosting those hot experts become stragglers while others idle. Three mitigations are standard in 2026 stacks:

Expert replication / redundant experts. Duplicate the hottest experts onto multiple GPUs so their load spreads. DeepSeek’s deployment adds “redundant experts” — extra copies of high-traffic experts — so no single GPU is a bottleneck.
Expert Parallel Load Balancer (EPLB). DeepSeek open-sourced EPLB, which computes an expert-to-GPU placement that equalises expected load, optionally packing frequently co-activated experts together to cut cross-node hops. You periodically recompute placement from observed routing statistics and reshuffle.
Grouped / hierarchical all-to-all. Route within a node first (over fast NVLink) and only cross node boundaries for the residual, cutting the volume that traverses slow inter-node links. DeepSeek-V3’s node-limited routing caps how many nodes a token’s experts can span, bounding cross-node traffic by design.

A capacity and throughput worked example

Consider a 256-expert, top-8 MoE served with expert parallelism across two 8-GPU nodes (EP=16, so 16 experts per GPU). Suppose a decode batch of 512 tokens. Total expert assignments this step are 512 * 8 = 4096. Spread perfectly, each of 256 experts sees 4096 / 256 = 16 tokens, and each GPU (16 experts) processes 16 * 16 = 256 tokens’ worth of FFN. Now assume real routing skew where the busiest GPU sees 1.6x the average: it processes ~410 token-experts while the idlest sees ~150. The step is gated by the 410 straggler, so ~37% of your aggregate FFN capacity is wasted to imbalance. Enable EPLB plus one redundant copy of the top-4 hottest experts and the skew typically falls toward 1.15x, recovering most of that headroom.

Now the communication side. Each dispatched token moves a hidden vector — say 7168 dims at BF16, ~14KB. At 4096 assignments that is ~57MB dispatched and ~57MB combined per layer, per step. Over 60 layers that is roughly 6.8GB of all-to-all traffic to emit one decode token across the cluster. On a 400GB/s effective inter-node fabric that is ~17ms of raw transfer if unoverlapped — which is why hiding it behind compute is not optional. The decision matrix below summarises how the levers trade off.

The same arithmetic explains why prefill and decode behave so differently. In prefill you process, say, 4,000 prompt tokens at once, so a single all-to-all carries 4,000x more payload than a decode step — it is bandwidth-bound and the fixed launch overhead is negligible, so wide expert parallelism amortises beautifully. In decode you emit one token per sequence, the payload per collective is tiny, and the millisecond-scale launch and synchronisation overhead of the collective itself dominates. That is the structural reason a decode step’s MoE layer can be latency-bound on communication even when the GPUs are barely doing any floating-point work, and why the two phases deserve different EP widths.

Overlapping communication with compute

The reason DeepEP and similar libraries matter is that a naive MoE layer serialises three phases — dispatch, expert FFN, combine — and stalls the GPU on the network twice per layer. The fix is pipelining: while the dispatch for one micro-batch is in flight, the GPU runs attention or the expert FFN of another, so the all-to-all latency hides behind useful math. DeepEP ships two flavours of kernel for exactly this reason: a high-throughput set tuned for prefill’s large, bandwidth-bound transfers, and a low-latency set for decode that minimises per-collective overhead and can overlap with compute so tightly that the network cost nearly disappears from the critical path. On multi-node deployments this overlap is frequently the difference between a decode step gated by FLOPs and one gated by round-trips — a 2x swing in tokens per second is common. If you take one implementation lesson from this article, it is that comms-compute overlap is not a nice-to-have optimisation for large MoE serving; it is the thing that makes cross-node expert parallelism viable at all.

Lever	Helps	Costs	Use when
Higher capacity factor	Fewer dropped tokens, better quality	More padding + comms	Quality-sensitive, skewed batches
Expert replication (redundant experts)	Cuts straggler load	Extra HBM per copy	A few dominant hot experts
EPLB placement	Balances load cluster-wide	Recompute + reshuffle overhead	Stable traffic patterns
Node-limited / grouped routing	Less slow inter-node traffic	Slight quality constraint	Multi-node EP over IB/Ethernet
Expert offload to host	Fits big model on few GPUs	Latency from PCIe fetches	Memory-bound, latency-tolerant
Comms/compute overlap (DeepEP-style)	Hides all-to-all latency	Kernel complexity	Always, for multi-node decode

Trade-offs, Gotchas, and What Goes Wrong

MoE serving fails in ways dense serving never does, and most incidents trace back to the router and the collectives rather than the math.

Figure 4: Disaggregating prefill and decode lets each pool pick its own EP width, while an expert load balancer keeps hot experts from bottlenecking either pool.

Long description: a client request hits a scheduler that sends it to a prefill pool running wide expert parallelism at high batch, the KV cache is handed off to a decode pool tuned for low latency, streamed tokens return to the client, and an expert load balancer replicates hot experts onto busy GPUs across both pools.

Prefill and decode want opposite things. Prefill processes thousands of tokens at once, so its all-to-all is large, bandwidth-bound, and amortises well — wide expert parallelism and big batches win. Decode emits one token per sequence, so its all-to-all is tiny, latency-bound, and dominated by round-trip overhead. Running both on the same EP configuration compromises both. This is exactly why prefill/decode disaggregation pairs so naturally with MoE: give the prefill pool a throughput-tuned EP layout and the decode pool a latency-tuned one, and move the KV cache between them.

Load imbalance shows up as tail latency, not errors. Everything “works,” but p99 balloons because one GPU’s hot expert gates every step. Teams chase phantom network problems when the fix is EPLB or a redundant expert. Always export per-expert token counts and per-GPU MoE-layer time as first-class metrics.

Token dropping silently degrades quality. A capacity factor tuned for throughput can drop tokens on skewed prompts, and you will only see it as a mysterious quality regression on certain inputs, never as a crash. Prefer drop-free decode.

Small decode batches starve the experts. With few concurrent requests, each expert sees a handful of tokens, the FFN matmuls are tiny and inefficient, and the fixed all-to-all overhead dominates. MoE throughput is far more batch-sensitive than dense throughput; under light load a big MoE model can be startlingly slow per token. Continuous batching is not optional here.

KV cache is still dense. MoE sparsifies the FFN, not attention. The KV cache grows with batch and context exactly as in a dense model, and on long-context workloads it, not the experts, becomes your memory ceiling — so KV cache optimization matters just as much for MoE as for dense serving.

Practical Recommendations

Start by separating the two axes of scale: use expert parallelism for the routed experts and tensor parallelism for attention and any shared/dense experts, and only add pipeline parallelism when depth forces it. Match your EP degree to the expert count so each GPU owns a clean integer number of experts. Keep expert parallelism inside a single high-bandwidth domain (NVLink/NVSwitch) whenever the model fits; cross-node EP is where all-to-all starts to hurt, so reach for grouped/hierarchical routing and comms-compute overlap before you span nodes.

If you serve a large MoE on constrained hardware, expert offload to host memory lets you fit the model by keeping cold experts in CPU RAM and fetching them over PCIe on demand — viable for latency-tolerant or single-stream workloads, painful for tight interactive SLAs. For high-QPS serving, disaggregate prefill and decode and tune each pool’s EP independently.

Actionable checklist:

Set EP degree so experts divide evenly across GPUs; keep EP within one NVLink domain if it fits.
Use drop-free or dynamic capacity for decode; reserve tight capacity factors for throughput-only offline jobs.
Turn on comms-compute overlap (DeepEP-style kernels) for any multi-node deployment.
Export per-expert token counts and per-GPU MoE-layer latency; alert on skew above ~1.3x.
Enable EPLB and replicate the top few hot experts once you see stable skew.
Disaggregate prefill/decode and give each its own EP layout under real load.
Budget KV cache memory separately — MoE does not shrink it.
Rely on continuous batching; never benchmark MoE at batch size 1 and extrapolate.

Frequently Asked Questions

How is expert parallelism different from tensor parallelism?

Tensor parallelism splits every weight matrix across GPUs so each does a fraction of every matmul, synchronising with all-reduce sized to the full hidden dimension. Expert parallelism gives each GPU whole experts, so a GPU only computes for tokens routed to its experts, synchronising with all-to-all sized to the routed tokens. TP suits dense layers and attention; EP suits the sparse FFN. Production stacks compose both — TP for attention, EP for routed experts — because their communication patterns and cost drivers are fundamentally different.

Why does the all-to-all communication dominate MoE inference latency?

Each MoE layer issues two all-to-all collectives, dispatch and combine, and a 60-layer model does 120 of them per generated token. All-to-all is synchronising — the slowest transfer gates the layer — and during decode the batches are tiny, so fixed network round-trip overhead dwarfs the actual FFN math. Across nodes over InfiniBand or Ethernet this becomes the primary cost, which is why libraries like DeepEP exist purely to accelerate and overlap dispatch and combine with compute.

What is token dropping and does it hurt quality?

To keep tensor shapes fixed, each expert has a capacity limit. When routing sends more tokens to a hot expert than its capacity, the overflow is dropped (passed through the residual, skipping the FFN) or rerouted. Dropping keeps memory bounded and shapes static but silently degrades quality on skewed inputs. Most 2026 inference stacks prefer drop-free or dynamic-capacity decode, accepting some wasted padding compute in exchange for not corrupting outputs on unlucky batches.

What is EPLB and when should I use it?

EPLB — Expert Parallel Load Balancer, open-sourced by DeepSeek — computes an expert-to-GPU placement that equalises expected per-GPU load from observed routing statistics, and can co-locate frequently co-activated experts to cut cross-node hops. Use it once traffic patterns are stable enough that a recomputed placement stays valid for a while. It pairs with redundant (replicated) hot experts. If your skew is dynamic or bursty, favour replication and grouped routing over frequent EPLB reshuffles.

Can I serve a huge MoE model on a single GPU?

Sometimes, via expert offload. Cold experts live in CPU host memory and are fetched over PCIe when a token routes to them, keeping only the active working set in HBM. This fits models far larger than your GPU, at the cost of PCIe-fetch latency whenever a needed expert is not cached. It suits offline, batch, or latency-tolerant single-stream workloads. For interactive SLAs, a proper multi-GPU expert-parallel deployment with an in-HBM expert cache is far better.

Does MoE reduce KV cache memory?

No. MoE sparsifies only the feed-forward blocks; attention stays dense, so the KV cache grows with batch size and sequence length exactly as in a dense model of the same hidden size. On long-context workloads the KV cache, not the expert weights, often becomes the binding memory constraint. Treat KV cache capacity as an independent budget and apply the usual optimisations — paging, quantization, and compression — regardless of how sparse the FFN is.

Expert-Parallel MoE Inference: Serving Sparse Models at Scale (2026)