Expert Parallelism LLM Inference: Tensor vs Expert vs Data Parallelism for MoE Serving

Most teams that move from a dense model to a Mixture-of-Experts (MoE) model keep the parallelism recipe that worked before: tensor parallelism across the eight GPUs of a node, replicate the node for scale. It works, and it is often leaving a large share of the hardware’s throughput unused. A sparse model with 256 routed experts and only eight active per token has a very different cost structure from a dense transformer, and the parallelism strategy that matches it is different too.

This matters now because the open-weight frontier is MoE almost without exception, and the serving engines have caught up. vLLM and SGLang both ship expert parallelism for LLM inference as a first-class mode, with specialised all-to-all kernels, load balancers, and prefill/decode disaggregation. The gap between a naive tensor-parallel deployment and a tuned wide-EP deployment is reported in the multiple-x range on the same GPUs.

You will leave with a mechanism-level model of what each strategy costs in communication, memory and batch efficiency, a decision procedure, and concrete launch configurations.

What this covers: the three strategies and what each shards, the communication arithmetic, how real engines combine them (DP attention plus EP), load balancing, failure modes, and a decision matrix.

Context and Background

LLM inference has two phases with opposite bottlenecks. Prefill processes the whole prompt in parallel and is compute-bound. Decode generates one token per sequence per step and is memory-bandwidth-bound, because every step must stream the active weights and the growing key-value (KV) cache from HBM. Parallelism strategies mostly exist to fight one of those two bottlenecks, or the capacity limit that comes before them: the model does not fit on one GPU.

Tensor parallelism (TP) splits individual weight matrices across GPUs. Each layer’s matmuls are computed in shards and results are combined with all-reduce collectives, in the Megatron-LM style. Data parallelism (DP), in inference, means running independent replicas that each handle a subset of requests; no weights are exchanged between them. Expert parallelism (EP) is specific to MoE: whole experts are placed on different GPUs and tokens are routed to wherever their chosen experts live.

MoE changes the arithmetic. In DeepSeek-V3, described in the technical report on arXiv (2412.19437), the model has 671B total parameters but about 37B activated per token, with 256 routed experts per MoE layer plus a shared expert, and eight routed experts selected per token. Memory has to hold all 671B parameters, yet each token only pays compute and weight-read cost for a small slice. That mismatch is the entire opportunity and the entire difficulty.

The incumbent practice is TP inside a node, sometimes with pipeline parallelism across nodes. It is simple and well supported. Its weakness for sparse models is that sharding every expert matrix across all GPUs forces an all-reduce per layer while giving each GPU a thin slice of every expert, which makes the grouped matrix multiplications small and inefficient.

If you are comparing engines before choosing, our LLM inference benchmark of vLLM, TGI, SGLang and Triton covers the framework layer. This post stays at the parallelism-design layer: what you shard, and why.

A word on sources. Throughput figures below come from two published deployments, DeepSeek’s own inference system overview and the LMSYS large-scale EP write-up. Flags and defaults come from the vLLM expert parallel deployment docs. Arithmetic that I derive myself is labelled as illustrative.

The Core Argument: Shard Experts, Replicate Attention

The direct answer: for sparse MoE models at high traffic, expert parallelism combined with data-parallel attention usually beats pure tensor parallelism, because it keeps expert matmuls large, avoids duplicating KV cache, and replaces per-layer all-reduces with point-to-point token routing. Tensor parallelism still wins at low batch sizes, on small models, and on slow interconnects.

Figure 1: The reference layout. Attention runs data-parallel on every rank with its own KV cache. Expert weights are partitioned across the same GPUs, and an all-to-all moves token activations to the owning experts and back.

Figure 1 shows the shape. Every GPU hosts two roles. In the attention role it holds a replica of the attention weights and serves its own share of requests, with its own KV cache. In the expert role it holds a disjoint slice of the experts. Between the two, two all-to-all exchanges per MoE layer, called dispatch and combine, move activations. The load balancer in front is an ordinary request router.

What each strategy actually shards

Tensor parallelism shards every matrix. For an MLP block, the first projection is split by columns and the second by rows, so each GPU produces a partial sum and an all-reduce completes the layer. A transformer layer needs two all-reduces in the forward pass, one after attention output projection and one after the MLP. Cost scales with the number of tokens in flight times the hidden size, independent of how sparse the MLP is.

Data parallelism shards nothing in the weights. It shards the request stream. Each replica needs the full model, so on its own DP only scales throughput when the model already fits on one device or one TP group. Its virtue is zero inter-replica communication on the critical path.

Expert parallelism shards the expert set. With 256 routed experts and an EP size of 32, each rank holds 8 routed experts. Nothing is split inside an expert, so each expert’s matmuls keep their full shape. That is the key difference from TP: sharding by whole experts preserves arithmetic intensity, while sharding every expert’s columns shrinks it.

Why the combination is DP attention plus EP

Attention is not sparse. If you run TP for attention across many GPUs, you shard heads and the KV cache. That is fine for standard multi-head or grouped-query attention. But DeepSeek-style Multi-head Latent Attention (MLA) compresses the KV cache into a single low-rank latent per token, which cannot be sharded across heads the same way, so TP ends up duplicating the KV cache on each TP rank. The LMSYS write-up notes that data-parallel attention eliminates this KV duplication, which is why SGLang introduced DP attention in v0.4.

So the practical pattern is: attention data-parallel (each rank owns requests and their KV), MoE expert-parallel across the same ranks. DeepSeek’s production system uses exactly this: per their overview, “Routed Expert EP32 and MLA/Shared Expert DP32” for prefill and EP144 with DP144 for decode.

vLLM encodes the same coupling in its sizing rule. Per the docs, EP size is computed automatically as EP_SIZE = TP_SIZE × DP_SIZE. With TP of 1 the attention weights are replicated across DP ranks; with TP above 1, attention is sharded across TP ranks within each DP group. Without the --enable-expert-parallel flag, MoE layers use tensor parallelism of size TP×DP instead. The same GPUs, the same flags minus one, and a completely different communication pattern.

Why sparsity demands a large batch

DeepSeek’s overview makes a point that is easy to miss: high sparsity requires an extremely large overall batch. With 8 of 256 experts active, a batch of 256 tokens routes on average only 8 tokens to each expert, assuming perfectly uniform routing. A matmul with 8 rows is deep in memory-bound territory: you read the whole expert weight matrix to do almost no work.

An illustrative calculation (my arithmetic, not a measurement): with top-8 routing over 256 experts, the expected tokens per expert is batch × 8 / 256 = batch / 32. To give each expert even 64 tokens, which is roughly where grouped GEMMs on current GPUs start to approach compute-bound, you need a global batch of about 2,048 tokens in flight for that layer. Decode with 128 sequences per GPU across 32 GPUs reaches 4,096 tokens globally, comfortably past that. A single 8-GPU TP replica serving a few dozen concurrent sequences does not.

This is the central argument for wide EP: pooling the batch across many GPUs makes each expert see enough tokens to be efficient, and spreading experts means each GPU reads fewer expert weights per step. DeepSeek states both effects directly: EP scales the batch size, improving matmul efficiency, and each GPU handling fewer experts reduces memory access demand and lowers latency.

Deeper Analysis: Communication, Overlap and Load Balance

Figure 2: One MoE layer under EP. Router output decides destinations; dispatch sends token copies (FP8 in DeepSeek’s system), experts run grouped GEMMs, and combine returns BF16 outputs for a weighted sum.

Figure 2 is the per-layer critical path. Cost lives in the two arrows labelled all-to-all, so the next question is how large they are compared with TP’s all-reduce.

Communication volume: an illustrative comparison

Take DeepSeek-V3’s hidden size of 7,168 (from the technical report) and a rank holding T = 4,096 tokens in a prefill batch. These numbers are an arithmetic illustration, not a benchmark.

TP all-reduce at BF16: each all-reduce handles T × 7,168 × 2 bytes ≈ 58.7 MB of activations. A ring all-reduce across N GPUs moves roughly 2(N−1)/N times that per GPU, about 103 MB for N = 8. Two per layer gives roughly 206 MB per GPU per layer, and this cost is paid in every layer, dense or sparse.

EP dispatch: each token is copied to up to 8 experts. Worst case, T × 8 × 7,168 bytes at FP8 is about 235 MB per rank sent, per MoE layer. Combine at BF16 returns about 470 MB worst case. This looks larger than TP, and in raw bytes it can be. Three things change the picture.

First, copies to experts on the same GPU never cross a link. Second, DeepEP, the library that DeepSeek released for this, de-duplicates traffic per destination node in its internode mode, so a token routed to several experts on one remote node crosses the network once. Third, EP traffic is point-to-point over whatever fabric exists, and it overlaps with computation, while a TP all-reduce on the same data sits on the dependency chain.

The honest conclusion: EP does not necessarily move fewer bytes. It moves them in a pattern that can be hidden behind compute and that scales with top-k rather than with the TP group size. TP’s cost grows as you widen the group, since more GPUs means more all-reduce participants and smaller shards. EP’s cost per token stays roughly constant as you add ranks. That is why wide EP scales and wide TP does not.

Two kernel modes for two phases

The DeepEP library offers two dispatch modes, as the LMSYS article describes. Normal dispatch is built for long inputs such as prefill. It maximises throughput but produces symbolic (data-dependent) shapes, so it cannot be captured in a CUDA graph. Low-latency dispatch is built for decode. It supports CUDA graphs but needs a preallocated, fixed memory size; exceeding it causes a runtime error.

That asymmetry has a design consequence. Without prefill/decode (PD) disaggregation, an engine cannot use both modes in one communication group, which LMSYS reports makes the auto mode incompatible with DP attention. PD disaggregation removes the conflict: prefill instances use normal dispatch, decode instances use low-latency dispatch, both under DP attention.

vLLM mirrors this in its backend flag. --all2all-backend accepts allgather_reducescatter (the default, which works with any EP plus DP configuration), deepep_high_throughput for multi-node prefill, deepep_low_latency for multi-node decode with CUDA graph support, and two FlashInfer NVLink backends for multi-node NVLink (MNNVL) systems. The docs also list a combined prefill and decode pattern where the two instances use different backends and exchange KV cache through the NIXL connector.

For related background on how the engine’s execution loop interacts with these choices, see our post on the vLLM 0.30 Model Runner V2 migration.

Overlapping communication with compute

All-to-all latency is the tax on EP, and the engines pay it down by overlap. DeepSeek’s system splits a prefill batch into two microbatches that alternate, so one microbatch’s communication hides behind the other’s computation. For decode, where one microbatch is too small to split usefully, the attention layer is divided into two steps and run through a five-stage pipeline to the same end.

SGLang implements the prefill version as two-batch overlap (TBO), and vLLM exposes dual-batch overlap with --enable-dbo. The LMSYS numbers show where the technique pays off and where it hurts:

TBO result (SGLang, DeepSeek-V3)	Reported figure
Prefill throughput gain at same tokens per device	27% to 35%
Prefill max tokens per device: vanilla vs TBO	OOM above 8,192 vs 16,384 works
Decode speedup at 256 tokens per device	25.5% (22,310 tokens/s)
Decode at 32 tokens per device	−27% (slower)
Decode with simulated MTP, 128 sequences per device	35% (17,552 vs 12,929 tokens/s)

The last decode row matters operationally. Overlap needs enough work in each microbatch to be worth the splitting overhead; LMSYS found gains minimal or negative below a threshold of roughly 64 to 128 per device. If your traffic does not produce that batch, turn overlap off.

Load imbalance and EPLB

Uniform routing is a fiction. Real routers send far more tokens to some experts than others, and an all-to-all finishes only when the slowest rank finishes. One hot expert on one GPU makes every GPU wait.

The mitigation is the Expert Parallelism Load Balancer (EPLB). It collects expert load statistics, then computes a placement that evens out per-GPU load, optionally duplicating hot experts into redundant slots. LMSYS reports that adding 32 redundant experts on top of 256, for a pool of 288, also unlocks parallel sizes such as 12 or 72 that do not divide 256. In their ablation, EPLB gave a 1.49x speedup for prefill and 2.54x for decode.

In vLLM, enable it with --enable-eplb and configure it with --eplb-config. From the docs, the defaults are window_size 1000 steps, step_interval 3000 steps, num_redundant_experts 0, use_async true and policy “default”. The docs recommend 32 redundant experts for large-scale use, and quantify the cost for DeepSeek-V3 at roughly 2.4 GB of memory with one redundant expert per EP rank. That memory comes out of your KV cache budget, which brings us to trade-offs.

DeepSeek’s production system runs three separate balancers, which is a useful model for thinking about load: the prefill balancer evens attention compute and token counts per GPU, the decode balancer evens KV cache use and request counts, and the expert-parallel balancer minimises the maximum dispatch receive load across GPUs.

Prefill/decode disaggregation as the enabling move

Wide EP is hard to run well on a monolithic engine because prefill and decode want different kernels, different batch shapes, and different parallel sizes. DeepSeek uses EP32 for prefill and EP144 for decode. The reasons are visible in the numbers: prefill is compute-heavy and needs fewer GPUs per unit of work, while decode needs a very large EP group to push per-GPU expert count down to two routed experts and pool a large batch.

Figure 3: Disaggregated serving topology. A router sends requests to a prefill pool using high-throughput dispatch; KV cache moves to a decode pool using low-latency dispatch; EPLB reads load statistics and rebalances both.

In vLLM, the prefill instance runs deepep_high_throughput, the decode instance runs deepep_low_latency, and KV cache moves through the NIXL connector configured as {"kv_connector":"NixlConnector","kv_role":"kv_both"} on both. The documented client flow sends the prefill request with max_tokens=1 and a remote-decode flag, then passes the returned transfer parameters to the decode instance under the same request ID. The docs note the prompt must exceed the 16-token block size.

Published results

Two public data points ground the discussion. DeepSeek reported, for a 24-hour window in February 2025 on H800 GPUs, an average of about 73.7k input tokens per second per node during prefill (including cache hits) and about 14.8k output tokens per second per node during decode. Average occupancy was 226.75 nodes of 8 GPUs, with a stated cost of $87,072 per day at an assumed $2 per GPU-hour. Those are DeepSeek’s own accounting and assumptions, and the page itself says real revenue is far below the theoretical figure.

LMSYS reproduced a similar design on 12 nodes of 8 H100 each (96 GPUs). Per node, they report 52.3k input and 22.3k output tokens per second on 2,000-token inputs, and an estimated local cost of $0.20 per million output tokens. Against their own baseline of SGLang with TP16 groups plus DP attention, they report up to 3.3x for prefill (4 nodes, EP32) and 5.2x for decode (9 nodes, EP72), and up to 5x output throughput overall.

Read those comparisons carefully. The baseline is a reasonable TP deployment, not a strawman, but it is also one vendor’s tuned comparison on one model at one input length. Treat “up to 5x” as an existence proof that the gap can be large, not as what you will get. Your routing skew, input lengths and fabric will move it.

Choosing: A Comparison of the Three Strategies

Figure 4: The decision path from model and cluster to a parallelism layout. The final leaf is the wide-EP, PD-disaggregated design discussed above.

The following matrix summarises the trade-offs. The qualitative ratings are my synthesis of the mechanisms above, not benchmark outputs.

Dimension	Tensor parallel	Data parallel	Expert parallel
What is sharded	Every weight matrix	Request stream only	Whole experts
Collective	All-reduce, two per layer	None on critical path	All-to-all, dispatch and combine
Scales with	Group size (worse as it grows)	Replica count (linear)	Rank count (roughly flat per token)
Best phase	Low-latency decode, small batch	Anything that fits	Large-batch prefill and decode
Expert GEMM shape	Thin slices, low efficiency	Full model per replica	Full experts, high efficiency
KV cache	Sharded for GQA, duplicated for MLA	Per replica	Pair with DP attention
Interconnect need	NVLink class	Almost none	NVLink plus fast RDMA across nodes
Failure blast radius	One TP group	One replica	Whole EP group
Load imbalance	None	Request level	Expert level, needs EPLB
Operational complexity	Low	Lowest	High

When tensor parallelism is still right

TP is the correct default in several situations. If the model is small enough that its experts fit on one node, and traffic is modest, a single node with TP gives the lowest latency per request, since there are no cross-node hops and no load-imbalance tail. If your batch per replica rarely exceeds a few dozen sequences, EP’s batch pooling has nothing to pool. And on hardware without a fast cross-node fabric, wide EP will be bottlenecked on its all-to-all. The vLLM docs themselves say the DeepEP kernels may perform poorly on mixed workloads and suggest a smaller model or multi-node setup for H100 nodes.

For a compact MoE the picture is similar. Our explainer on the Bilibili Index Translate 35B-A3B MoE is a good example of a model with a small active footprint: it fits comfortably on a single node, so the interesting decision is DP replication count, not wide EP.

When pure DP is right

If the model fits on one GPU or one small TP group, and you only need more throughput, replicate. DP has no communication, trivially balances by request, and fails gracefully. The only reason to leave it is memory: you cannot fit, or you cannot afford to hold N copies of the weights when the weights dominate the budget.

When EP wins

EP wins when four conditions hold together: the model is genuinely sparse with many experts, aggregate traffic is high enough to fill large batches, there is a fabric that can carry all-to-all across nodes, and the team can operate a more complex system. DeepSeek’s V3/R1 design is the canonical case, with 256 routed experts, top-8 routing, and a fleet-scale workload.

A useful rule of thumb, which is my heuristic and not a standard: if tokens per expert per step at your realistic peak batch is below roughly 16 to 32, you are memory-bound on expert weights, and pooling more batch via EP is likely to help. If it is already above 128, you are compute-bound and the extra complexity buys less.

Practical Launch Configurations

The following commands use the flags documented by vLLM. They are starting points, not tuned configurations; verify flag names against your installed version, since these interfaces have been changing quickly.

Single node, EP with DP attention

The vLLM docs give this form for an 8-GPU node:

vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --tensor-parallel-size 1 \
    --data-parallel-size 8 \
    --enable-expert-parallel

With TP of 1 and DP of 8, EP size is 8: each GPU holds a replica of attention and 32 of the 256 routed experts. Remove --enable-expert-parallel and the same command turns the MoE layers into TP of size 8 instead, which is the cleanest A/B test you can run on a node.

Two nodes, wide EP with low-latency kernels

# Node 1 (primary)
vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --all2all-backend deepep_low_latency \
    --tensor-parallel-size 1 \
    --enable-expert-parallel \
    --data-parallel-size 16 \
    --data-parallel-size-local 8 \
    --data-parallel-address 192.168.1.100 \
    --data-parallel-rpc-port 13345 \
    --api-server-count=8

# Node 2 (secondary, no API server)
vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --all2all-backend deepep_low_latency \
    --tensor-parallel-size 1 \
    --enable-expert-parallel \
    --data-parallel-size 16 \
    --data-parallel-size-local 8 \
    --data-parallel-start-rank 8 \
    --data-parallel-address 192.168.1.100 \
    --data-parallel-rpc-port 13345 \
    --headless

Here --data-parallel-size is the global count and --data-parallel-size-local the per-node count; the secondary node sets its start rank to the cumulative local DP of previous nodes. Prerequisites per the docs: install DeepEP and DeepGEMM, and gdrcopy for disaggregated serving. On InfiniBand clusters, set GLOO_SOCKET_IFNAME to an Ethernet interface to avoid initialisation hangs.

Enabling load balancing and overlap

    --enable-eplb \
    --eplb-config '{"window_size":1000,"step_interval":3000,"num_redundant_experts":32,"log_balancedness":true}' \
    --enable-dbo

Start with log_balancedness on and look at the numbers before trusting EPLB; if your routing is already balanced, redundant experts only cost memory. To benchmark without real traffic, the docs describe VLLM_MOE_ROUTING_SIMULATION_STRATEGY=uniform_random with VLLM_RANDOMIZE_DP_DUMMY_INPUTS=1. Be aware that this tests the best case for routing balance, so it overstates what real traffic will give you.

A back-of-envelope sizing helper

This small script (illustrative arithmetic only) estimates experts per rank and tokens per expert, which are the two numbers that tell you whether EP is paying off.

def ep_sizing(n_experts=256, top_k=8, ep_size=32, redundant=32,
              tokens_per_rank=128):
    pool = n_experts + redundant
    slots_per_rank = pool / ep_size
    global_tokens = tokens_per_rank * ep_size
    tokens_per_expert = global_tokens * top_k / n_experts
    return slots_per_rank, tokens_per_expert

for ep in (8, 16, 32, 72, 144):
    slots, tpe = ep_sizing(ep_size=ep, redundant=32 if ep > 8 else 0)
    print(f"EP{ep:>3}: {slots:5.1f} expert slots/GPU, "
          f"{tpe:6.1f} tokens/expert/step")

With 128 decode tokens per rank, tokens per expert per step grow linearly with EP size: about 32 at EP8, 128 at EP32, and 576 at EP144, assuming uniform routing. The slot count falls to 2 per GPU at EP144, matching the “2 routed experts per GPU” in DeepSeek’s decode configuration.

Trade-offs, Gotchas, and What Goes Wrong

The blast radius is the whole EP group. In a TP replica, a GPU failure loses one replica. In a 144-rank EP group where every rank participates in every MoE layer, one failed rank stalls all of them. Production wide-EP systems need fast detection, drain procedures, and spare capacity to re-form groups. If your orchestration cannot restart a group in seconds, the availability maths may favour smaller groups.

Stragglers set the speed. The all-to-all is a barrier. One slow GPU, one throttled link, or one hot expert delays everyone. EPLB addresses routing skew, not hardware variance; monitor per-rank step times, not just aggregate throughput.

Redundant experts cost KV cache. The docs quantify about 2.4 GB for DeepSeek-V3 with one redundant expert per rank, and recommend 32 redundant experts at large scale. On a memory-tight GPU that is context length you no longer have. This is why the docs caution EPLB may not suit memory-constrained setups.

Fixed buffers fail loudly. Low-latency dispatch needs preallocated memory sized in advance; the LMSYS write-up notes that exceeding it causes a runtime error. Size it for your peak tokens per rank, with margin, and test the worst-case burst, not the average.

Small batches defeat the design. Both overlap and batch pooling need volume. LMSYS measured a 27% slowdown from TBO at 32 tokens per device in decode. A wide EP deployment serving low traffic is expensive and slower than a TP node. Autoscaling is awkward too, because EP groups scale in large units.

Networking is a hard dependency. The vLLM troubleshooting notes list real failures: ulimit -l must be unlimited on InfiniBand or RoCE, an IBGDA driver script may need running, and Kubernetes pods may need hostNetwork: true and privileged mode to avoid NVSHMEM peer disconnects. These are the signs that EP sits closer to the hardware than TP does.

Hardware and engine support is uneven. The LMSYS article states support for NVIDIA Hopper only at the time of writing, with Blackwell in progress. Feature availability changes quickly across engine versions; I did not verify current Blackwell status for this post, so check release notes before planning a 2026 hardware purchase around it.

Precision interacts with communication. DeepSeek runs dispatch in FP8 and combine in BF16, keeping core MLA computation and combine transmissions in BF16 for quality. Halving dispatch bytes is a real saving, but it is a model-quality decision, not just a performance flag. Validate accuracy on your own evaluations.

Benchmarks lie by omission. Uniform-random routing simulators, fixed input lengths, and warmed prefix caches all flatter EP. DeepSeek’s own 73.7k figure includes cache hits. Compare like with like before quoting a number to your finance team.

Practical Recommendations

Start from the model, not the technique. If it is dense, stay with TP (plus pipeline parallelism if it spans nodes) and replicate with DP. If it is a sparse MoE, ask three questions in order: does it fit on a node, how big is your realistic steady-state batch, and what is your cross-node fabric?

For a small or medium MoE that fits on a node and serves moderate traffic, run TP or EP inside the node and scale with DP replicas. Use the --enable-expert-parallel A/B on a single node first. It costs a flag, and if your batch is large enough you will see the difference.

For a large MoE with heavy traffic, plan for wide EP with DP attention, but only after you can run it in a disaggregated form. Separate prefill and decode pools, give each its own all-to-all backend, and treat EPLB as a measured addition, not a default. Budget engineering time for the failure handling, because that is where most wide-EP projects spend their effort.

Quick checklist:

Count tokens per expert per step at your peak batch before deciding. Below about 16 to 32, EP is likely to help.
A/B the same hardware with and without --enable-expert-parallel on real traffic traces.
Pick DeepEP backends by phase: high-throughput for prefill, low-latency for decode.
Turn on log_balancedness and read it before enabling redundant experts.
Disable batch overlap if your per-device batch sits below roughly 64.
Reserve KV cache memory for redundant experts; recompute your context budget.
Design failure handling for the whole group, not one GPU.
Test with skewed, bursty routing, not only uniform-random dummy inputs.
Re-read the engine docs at every upgrade; flags and defaults are moving.

Frequently Asked Questions

What is expert parallelism in LLM inference?

Expert parallelism places different experts of a Mixture-of-Experts model on different GPUs. After the router picks experts for each token, an all-to-all collective sends the token’s activations to the GPUs holding those experts, which compute and return results through a second all-to-all. Unlike tensor parallelism, no expert is split, so each expert’s matrix multiplications keep full size and efficiency.

Is expert parallelism better than tensor parallelism for MoE models?

It depends on batch size and interconnect. For sparse MoE models with high traffic and a fast cross-node fabric, published deployments by DeepSeek and LMSYS report large gains from wide EP with DP attention. For small models, low traffic, or single-node deployments, tensor parallelism often gives lower latency with far less operational complexity. Test both on your own traffic.

How does vLLM decide the expert parallel size?

vLLM computes it automatically as tensor-parallel size multiplied by data-parallel size, once you pass --enable-expert-parallel. With TP of 1 and DP of 8, EP size is 8. Without the flag, vLLM instead applies tensor parallelism of size TP times DP to the MoE layers. You do not set the EP size directly.

What is EPLB and do I need it?

EPLB is the Expert Parallelism Load Balancer. It tracks how many tokens each expert receives and rearranges, and optionally duplicates, experts across GPUs so no rank becomes a hotspot. You need it when routing is skewed and EP groups are large. LMSYS reported 1.49x prefill and 2.54x decode speedups in an ablation. It costs memory, so measure first.

Why separate prefill and decode for expert parallel serving?

The two phases need different all-to-all kernels. DeepEP’s normal mode suits long prefill inputs but cannot be captured in a CUDA graph, while its low-latency mode suits decode and supports CUDA graphs with preallocated buffers. Separate pools let each use its best kernel and its own EP size, with KV cache transferred between them, for example via NIXL in vLLM.

Can I combine tensor and expert parallelism?

Yes. In vLLM, setting TP above 1 together with DP shards attention across TP ranks within each DP group while experts spread across all TP times DP ranks. This is useful when attention weights or KV cache are too large to replicate per rank, or when you want lower per-request latency than pure DP attention provides, at the cost of added all-reduce traffic.

Tensor vs Expert vs Data Parallelism for LLM Inference: MoE Serving Design (2026)