XDP Load Balancer Design: Katran, Cilium and Maglev Hashing

A software load balancer that handles tens of millions of packets per second per host used to mean one thing: take the NIC away from the kernel and poll it from user space. That assumption is now a choice rather than a necessity. An XDP load balancer runs inside the network driver, before the kernel has allocated a single socket buffer, and it still leaves the NIC, the TCP stack and your monitoring tools under normal kernel control. Meta’s Katran forwards production traffic this way, and Cilium uses the same hook to accelerate Kubernetes services.

The catch is that “fast in the driver” only matters if the surrounding design is right. Connection tracking, consistent hashing, encapsulation, MTU, and what happens when a backend leaves the pool decide whether the system is stable or resets your users’ connections at 3 a.m.

This post walks through the mechanism end to end: where XDP sits in the packet path, how Katran and Cilium build a Layer-4 forwarder on it, how Maglev hashing keeps flows pinned, and what the cycle budget looks like at 10, 25 and 100 Gbps.

What this covers: XDP hook modes and verdicts, the Katran data path, Maglev table construction with a worked example, Cilium’s XDP and DSR mode, an XDP versus DPDK versus IPVS comparison, packet-rate arithmetic, and the failure modes that bite in production.

Context and Background

A Layer-4 load balancer makes one decision per connection: which backend receives this flow. It does not terminate TCP or parse HTTP. That narrow job is why it can be fast, and why the interesting engineering is in the data path rather than in features.

Historically there were three ways to build one. The first is the in-kernel route: Linux IPVS, built on netfilter hooks, keeps a connection table and rewrites packets inside the normal stack. The second is kernel bypass: Google’s Maglev, described at NSDI 2016, takes packets straight from the NIC into user space and forwards them without touching the kernel. DPDK-based balancers follow the same pattern. The third is hardware: smart NICs, programmable switches, or commercial appliances that do the lookup in silicon.

Each carries a cost. IPVS inherits the per-packet overhead of the full stack. Kernel bypass dedicates cores to busy polling, takes the NIC away from the kernel so tcpdump, iptables and the usual tooling stop seeing it, and requires you to reimplement anything the stack gave you for free. Hardware is fast but rigid and expensive to change.

XDP, the eXpress Data Path, is a fourth point in this space. The 2018 CoNEXT paper by Høiland-Jørgensen and colleagues describes it as a programmable hook in the driver that keeps the kernel in control of the hardware rather than moving the hardware out of the kernel. A program written in restricted C, compiled to eBPF bytecode and checked by the in-kernel verifier, runs on every received packet. In that paper’s evaluation, Katran’s XDP component reached 5.2 million packets per second (Mpps) on one core and 29.3 Mpps on six, against 1.2 and 7.3 Mpps for IPVS at the same core counts, a gain of roughly 4.3 times on the single-core case.

The practical consequence is a new default. If you already run Linux with a recent kernel, you can get most of the forwarding rate of kernel bypass without giving up the operating system. For readers who want the algorithmic background on how load-balancing policies differ, see our comparison of load balancing algorithms, power of two choices, and Maglev at L4 versus L7. This post concentrates on the data plane that executes those policies. For the primary sources, the Katran repository and the XDP paper are the best starting points.

A note on evidence. Where this post cites a number it comes from the Katran README, the Cilium documentation, the Maglev and XDP papers, or a small simulation I ran for illustration. Numbers marked “illustrative” are arithmetic or simulation, not benchmarks of any product.

How an XDP Load Balancer Works: Reference Architecture

An XDP load balancer is an eBPF program attached to a network interface’s receive path that classifies each packet, picks a backend using a connection table or consistent hash, encapsulates the packet toward that backend, and transmits it straight back out of the NIC with the XDP_TX verdict. It never allocates a socket buffer, so the per-packet cost stays in the low hundreds of nanoseconds.

Figure 1: Where the XDP hook sits in the receive path and what each verdict skips.

Figure 1 shows the receive path. After the NIC DMAs a frame into a ring buffer, the driver’s poll routine (NAPI) picks it up. On the normal path the driver then builds an sk_buff, the kernel’s large per-packet metadata structure, runs generic receive offload, passes through traffic control, netfilter and routing, and finally reaches a socket. An XDP program runs before any of that. It sees a pointer to the raw frame, a bounds pointer, and a small metadata struct, and it returns a verdict.

Hook modes: native, offload and generic

XDP can attach in three modes, and the difference matters more than any tuning knob.

Native (driver) mode runs the program inside the driver’s receive loop, on the page that holds the frame. This is the mode that delivers the headline rates, and it requires the driver to implement XDP support. Most drivers for 10 Gbps and faster NICs do on a recent kernel; the Cilium documentation says as much for its XDP acceleration feature.

Offload mode pushes the eBPF program into the NIC itself, so packets can be dropped or reflected without reaching host memory. It depends on specific hardware and a restricted subset of helpers and map types, so portability is limited.

Generic mode runs the program later, after the kernel has already allocated an sk_buff, and works on any interface. It is useful for development, for virtual devices and for testing on a laptop, but it loses most of the performance benefit. For any XDP load balancer, mode choice is the first decision. The Katran README states plainly that generic mode is supported with lower performance than driver mode. In iproute2 the three modes appear as xdpdrv, xdpoffload and xdpgeneric.

The five verdicts and what each means for a balancer

A program returns one of five actions. XDP_DROP discards the frame and recycles the page, which is the cheapest possible outcome and the basis of XDP-based DDoS filtering. XDP_ABORTED also drops, but signals a program error and fires a tracepoint. XDP_PASS hands the frame to the regular stack; a load balancer uses it for anything that is not addressed to a virtual IP, such as SSH to the host or health checks. XDP_TX bounces the frame back out of the interface it arrived on. XDP_REDIRECT sends it to another interface, another CPU, or a user-space socket through AF_XDP, with the destination looked up in a typed map.

For a Layer-4 balancer, XDP_TX is the workhorse. The program rewrites the packet in place, prepends an outer header, fixes the Ethernet addresses to point at the next hop, and returns XDP_TX. The frame goes back onto the wire from the same receive queue’s transmit counterpart, with no copy. Cilium’s NodePort acceleration uses the same pair of actions when it forwards to a remote backend.

The paper’s measurements put this in context. A no-op XDP program that drops every packet reached 24 Mpps on one core, against 43.5 Mpps for DPDK in the same test. Per-packet cost works out to 41.6 ns for XDP and 22.9 ns for DPDK. The authors attribute much of that 18.7 ns gap, about 67 cycles on a 3.6 GHz core, to driver function-call overhead, since the mlx5 driver made ten calls per packet at roughly 1.3 ns each. In other words, the gap is an implementation detail of the driver path, not a limit of the model.

Verifier, maps and why the program stays small

The in-kernel verifier is what makes it safe to run user-supplied code in the driver. It builds a control-flow graph, rejects unbounded loops and unreachable code, and requires that every packet access be preceded by a bounds check and every map lookup result be checked for null. The practical effect on a balancer is that the program is a short, branch-light pipeline: parse Ethernet, parse IP, parse the transport header, look up, encapsulate, transmit.

State lives in BPF maps, key-value stores that are shared between the eBPF program and a user-space control plane. Maps come in global and per-CPU variants. Per-CPU maps matter for a balancer because they let every receive queue’s CPU update its own connection table and counters with no locks and no cache-line bouncing, which is the property the Katran README leans on when it claims throughput that scales with the number of NIC receive queues. LRU hash maps give a fixed-size connection table that evicts old entries automatically.

Why packet rate is the real constraint for an XDP load balancer

Every XDP load balancer is bound by packets per second, not bits per second, because every packet costs roughly the same fixed amount of work regardless of payload. The relevant stress case is minimum-size frames. On Ethernet, a 64-byte frame occupies 84 bytes of wire time once you add the 8-byte preamble and 12-byte inter-frame gap.

The arithmetic is simple. At 10 Gbps that is 10e9 / (84 x 8) = 14.88 Mpps, or 67.2 ns per packet. At 25 Gbps it is 37.2 Mpps, or 26.9 ns. At 100 Gbps it is 148.8 Mpps, or 6.72 ns per packet. These are theoretical line rates for the smallest frames, and real traffic is a mix. Still, they set the envelope for how many cycles each queue can spend.

With RSS (receive side scaling) spreading flows across queues, the per-core budget scales with the number of cores. Take a 3 GHz core. A 10 Gbps port with all 14.88 Mpps landing on a single queue leaves about 200 cycles per packet. Spread across 8 queues, each core sees 1.86 Mpps and has roughly 1,600 cycles per packet. That is the reason balancers can be so compact: the cycle budget is generous once flows are distributed, and the hard part is the memory behavior of the lookup, not the instruction count. These figures are illustrative arithmetic, not measurements.

The Katran Data Path: Anatomy of an XDP Load Balancer

Katran, open-sourced by Meta, is the best-known XDP load balancer and is a C++ control-plane library plus a BPF forwarding program. Its design makes four decisions that together define the shape of a modern XDP load balancer: it forwards with IP-in-IP encapsulation, it answers with direct server return, it chooses backends with a Maglev-derived hash, and it keeps flow state in a fixed-size LRU table.

Figure 2: Routers spread a virtual IP across Katran nodes with ECMP; backends reply directly to clients.

Topology: ECMP in front, DSR behind

In the Maglev paper and in Katran alike, every balancer node announces the same virtual IP (VIP), and the upstream router spreads flows across the nodes with equal-cost multipath (ECMP). The balancer is therefore horizontally scalable and has no single front door. Adding a node means announcing the VIP from one more machine.

On the return path, Katran operates only in direct server return mode. The backend, configured with the VIP on a loopback-type interface, builds its reply with the VIP as the source and sends it to the client without passing back through the balancer. Since replies are usually much larger than requests, this removes most of the bytes from the balancer’s path, and it is the main reason one balancer can front far more bandwidth than it forwards.

Encapsulation: IPIP and the RSS trick

Rather than rewriting the destination address, Katran wraps the original packet in an outer IP header addressed to the chosen real server. The inner packet, with its original destination VIP and client source, is untouched, which is exactly what DSR needs. The README says Katran uses IPIP encapsulation, and it adds a detail worth copying: the outer source address is chosen per flow, so different flows carry different outer sources while packets inside a flow keep the same one. That lets the receiving host’s RSS hash, which sees only the outer header, spread tunneled flows across its own queues instead of piling them onto one core.

The cost of encapsulation is 20 bytes for IPv4 in IPv4. A 1,500-byte packet from the client becomes 1,520 bytes toward the backend, which is larger than the standard Ethernet payload. This is where MTU becomes a design constraint, and I come back to it in the failure-mode section.

Server selection: the two-level lookup

The control flow in Katran, summarized from its README and shown in Figure 3, is short.

Figure 3: Per-packet decision flow of an XDP load balancer in the Katran style.

First, the program checks whether the destination address and port match a configured VIP. If not, it returns XDP_PASS and the host handles the packet normally. If so, it looks up the flow in the connection table. A hit with a live backend means the packet goes to the same real server as before. A miss means a new flow, so the program hashes the five-tuple (source address, source port, destination address, destination port, protocol), indexes the consistent-hash table, stores the result in the connection table, and encapsulates.

The connection table exists for a subtle reason. If the hash ring never changed, you would not need it. But backends come and go, and every change moves some slots in the ring. The table pins established flows to the backend that accepted them, so a ring rebuild affects only new connections. The ring handles the case where the table has no entry.

The table has a fixed size, set at startup, with old entries evicted by LRU. That is a deliberate trade: bounded memory and constant-time operations in exchange for the possibility that a long-idle flow loses its pin. A flow that loses its pin falls back to the ring, which usually points to the same backend, and that fallback is the entire reason consistent hashing is in the design.

Stateless failover between balancer nodes

The README makes a claim that sets Katran apart from a simple NAT balancer. Server selection uses only packet header data and a deterministic hash, so separate Katran instances that share the same backend list and seed pick the same real server without exchanging state. You can restart or drain one instance, ECMP moves its flows to another node, and the second node, missing a table entry, computes the same backend from the ring.

This only holds if the ring is identical across nodes, which requires the same backend ordering, the same weights, and the same seed. If two nodes disagree because a configuration push landed on one first, flows that reroute during that window go to different backends and reset. State-free failover is therefore a property you must keep true operationally, not something the code guarantees by itself.

Maglev Hashing: Building the Ring and Measuring Its Disruption

Maglev hashing is a consistent-hashing scheme that fills a fixed-size lookup table so that each backend owns an almost equal share of slots, and so that removing one backend moves few of the other slots. Lookup is a single array index, which is why it suits a data path with a nanosecond budget.

Table construction, step by step

The Maglev paper’s construction, as summarized by Adrian Colyer, works as follows. The table size M must be prime, and it should be much larger than the number of backends N; the paper’s guidance is M greater than 100 times N, which keeps the difference in share between backends to at most about 1 percent. Each backend gets a permutation of the slots 0 to M-1, derived from two hashes of its name:

offset = hash1(name) mod M
skip = hash2(name) mod (M-1) + 1
the j-th preference is (offset + j x skip) mod M

Because M is prime and skip is between 1 and M-1, this walk visits every slot exactly once, which is why the table size has to be prime. Backends then take turns. In each round, every backend claims the first slot in its permutation that is still empty. The loop ends when all M slots are filled. Weighted backends are handled by giving heavier backends more turns per round; Katran’s README says it modifies Maglev specifically to support unequal weights.

A tiny worked example makes the mechanism concrete. Take M = 7 and three backends. In the paper’s own example, a backend with offset 3 and skip 4 gets the preference list 3, 0, 4, 1, 5, 2, 6. Using SHA-256 to derive offsets and skips for backends named b0, b1 and b2 (my own illustrative choice of hash), the finished table is: slot 0 to b1, slots 1 to 3 to b0, slot 4 to b1, slots 5 and 6 to b2. The split is 3, 2 and 2 slots, as even as seven slots among three backends allows.

Why 100 times N, shown with numbers

I ran a small simulation to see the effect of table size, using 100 backends and the construction above. These are illustrative results from my own code, not benchmarks of any product, and a single backend removal is a thin sample.

Table size M	Slots per backend (min to max)	Removing one backend: other slots that changed
251	2 to 3	7 of 251 (2.8%)
16,381	163 to 164	225 of 16,381 (1.4%)
65,537	655 to 656	350 of 65,537 (0.5%)

Two points stand out. Load balance is nearly perfect once M is a large multiple of N. And the collateral movement, slots that change owner even though their backend did not leave, shrinks as M grows. The slots that belonged to the removed backend must move by definition, roughly 1 percent here. The extra churn is Maglev’s price for its near-uniform distribution: it trades some disruption for flatness, unlike a classic ring with virtual nodes, which minimizes movement but balances less evenly.

Cilium exposes the same trade-off as a configuration option. Its documentation lists maglev.tableSize with a default of 16,381 and the permitted primes 251, 509, 1021, 2039, 4093, 8191, 16381, 32749, 65521 and 131071, and advises M well above 100 times the backend count. It says 16,381 suits about 160 backends and 65,521 about 650. It also warns that changing the table size can temporarily cause inconsistent backend selection, and that every agent must share the same maglev.hashSeed.

What consistent hashing does and does not buy you

Consistent hashing gives you stability across table rebuilds and across balancer nodes. It does not give you stability across backend changes for flows whose table entry has been lost. If a flow’s connection-table entry is evicted and the ring has since been rebuilt for a different backend set, that flow’s hash may now point somewhere else, and the backend that receives its next packet has no matching TCP state.

This is the key insight that the connection table and the ring are not alternatives. The ring is a fallback that is correct most of the time. The table is the authority for live flows. Sizing the table so that it rarely evicts active flows is therefore as important as choosing M.

Cilium: XDP, DSR and Maglev for Kubernetes Services

Cilium applies the same ideas inside a Kubernetes cluster. Its kube-proxy replacement implements Service load balancing in eBPF, and it uses XDP only on the north-south path: traffic entering a node for a NodePort, LoadBalancer or externalIPs service. For a deeper look at the rest of the stack, see our guide to Cilium, eBPF service mesh, networking, observability and security.

Two load-balancing layers

Cluster-internal traffic does not go through XDP at all. The Cilium documentation explains that the kube-proxy replacement depends on socket-level load balancing, which rewrites the destination address at connect, sendmsg and recvmsg time. A pod connecting to a service virtual IP gets the backend address substituted before a packet is ever built, so no per-packet translation is needed. For traffic arriving from outside, a program on the node’s physical interface performs the translation, and that is where XDP acceleration applies.

Enabling it is a single setting. loadBalancer.acceleration accepts disabled (the default), native, and best-effort, the last of which turns XDP on only for devices whose drivers support it. With multiple devices, native turns XDP on for all of them, so every driver must support it. The documentation notes that it works with DSR, SNAT and hybrid modes, and that packets redirected at the XDP layer do not show up in tcpdump, so you rely on Cilium’s own monitor and metrics. cilium-dbg status --verbose reports XDP Acceleration: Native when active.

DSR dispatch modes

By default Cilium uses SNAT, which keeps replies flowing back through the node that received the request. Setting loadBalancer.mode=dsr makes the backend reply directly to the client and preserves the client source address. How the original service address and port reach the backend depends on the dispatch mode.

Dispatch	Mechanism	Main constraint
Option	Service IP and port in an IPv4 option or IPv6 extension header	Reduces MTU; some networks drop packets carrying IP options
Geneve	Geneve encapsulation	Works with native routing or Geneve tunneling, not VXLAN
IPIP	IPIP or IP6IP6 encapsulation, set with `loadBalancer.dsrDispatch=ipip`	Native routing required; frontend and backend ports must match

A fourth setting, loadBalancer.mode=hybrid, uses DSR for TCP and SNAT for UDP, which avoids having to change the MTU. Per-service selection is possible with bpf.lbModeAnnotation=true. The choice of dispatch mode is the same trade-off Katran made: IPIP is the cleanest encapsulation but costs 20 bytes and imposes port symmetry.

Platform caveats that bite early

The Cilium documentation is candid about environments. On AWS with the ena driver it requires driver version 2.2.8 or later, an MTU of at most 3498, and channel counts at most half of Combined. On Azure it needs Accelerated Networking, kernel 5.6 or later, a ConnectX-4 Lx or newer NIC and LRO disabled. GCP is listed as unsupported for this feature. Some clouds require source and destination checking to be disabled for DSR to work at all. These are the kind of conditions that turn “enable XDP” into a week of debugging, so test on the exact instance type before committing.

XDP vs DPDK vs IPVS vs Hardware: A Decision Matrix

The XDP versus DPDK question is usually framed as speed against convenience. The published numbers say the speed gap is real but smaller than folklore suggests, and the convenience gap is larger. In the XDP paper’s single-core drop test, DPDK reached 43.5 Mpps against XDP’s 24 Mpps. In the same paper’s Katran comparison, the XDP balancer reached 5.2 Mpps on one core and 29.3 Mpps on six, while IPVS reached 1.2 and 7.3 Mpps.

Note what the second set of numbers says. A full balancer that does connection lookup, hashing and encapsulation is far below the raw drop rate on the same core: 5.2 Mpps is about 192 ns per packet, roughly four times the 41.6 ns of the empty program. The forwarding logic and its memory accesses dominate, not the hook. Six cores delivered 29.3 Mpps, about 4.9 Mpps per core, so scaling was good but not perfectly linear. These results come from one paper’s testbed and a particular Katran version; treat them as a shape, not a promise for your hardware.

Approach	Strength	Weakness	Pick it when
XDP (Katran, Cilium)	Kernel stays in control; tcpdump and stack still work for non-VIP traffic; no busy polling; scales with RX queues	Needs a capable driver; restricted program model; XDP_TX and DSR topology constraints	You run Linux at scale and want most of bypass speed with kernel tooling
DPDK or Maglev-style bypass	Highest per-core rate; full control of memory and batching	Dedicated polled cores; NIC removed from kernel; must reimplement stack features	You need the last 2x per core and can own a custom stack
IPVS	Mature, no eBPF needed, rich schedulers	Lower per-core rate in the XDP paper’s test; runs inside netfilter path	Moderate rates, existing operational knowledge
Hardware or smart NIC	Deterministic latency, offloads the CPU	Cost, rigid feature set, vendor lock-in	Fixed, very high rate policies with stable rules

Two qualifiers on this matrix. First, the paper’s forwarding latency table shows XDP forwarding averaging 82 microseconds at 100 packets per second and 7 microseconds at 1 Mpps, against 2 and 3 microseconds for DPDK. The cause is interrupt-driven receive versus polling, and it reflects power-saving behavior at low rates rather than a flaw in the hook. A balancer under real load sits at the favorable end of that curve. Second, the Katran README claims co-located applications pay no penalty relative to kernel-bypass approaches. That is the upside of not dedicating cores to polling, but it is a vendor claim without published figures in the README, so measure it in your environment.

A cycle-budget walk-through (illustrative)

Pull the pieces together with a back-of-envelope plan for a 25 Gbps host. Assume a 3 GHz core, a mixed traffic profile, and a target of handling the worst case of 37.2 Mpps of minimum-size frames.

If the balancer costs about 190 ns per packet per core, as the Katran result in the paper implies, one core handles about 5.2 Mpps. To reach 37.2 Mpps you need seven cores, or eight with headroom. At 100 Gbps the line-rate worst case of 148.8 Mpps would need roughly 29 cores at that per-packet cost, which tells you why 100 Gbps designs lean on larger average packet sizes, multiple balancer nodes behind ECMP, or both. The cost model is simple enough to keep in a spreadsheet: cores = line-rate Mpps divided by per-core Mpps, with a 20 to 30 percent margin for bursts and maintenance.

Where do the 190 ns go? A plausible breakdown, which I offer as reasoning and not as a profile, is a handful of cache misses in the connection table and the ring (each potentially tens of nanoseconds when the working set exceeds the cache), header parsing and checksum updates, the encapsulation write, and the transmit descriptor handoff. This explains why the table sizing and key layout matter more than shaving instructions. A connection table that fits in L2 or L3 turns misses into hits, and per-CPU maps keep the cache lines private.

Failure and capacity view of the whole system

Put the pieces into one picture of what can fail in an XDP load balancer deployment. A balancer node can die, in which case ECMP rehashes its flows onto the survivors and the ring, which is identical everywhere, routes them back to their backends. A backend can fail health checks and be removed, in which case the ring changes and only flows without a table entry are at risk. The router’s ECMP set can change, which reshuffles which balancer sees which flow. And the connection table can fill, which evicts live entries.

Each of these is survivable alone. The dangerous case is two at once, such as a node restart during a backend drain, because then flows lose their table entry and the ring has also changed. Plan maintenance so that these events are serialized, and you remove most of the exposure.

Trade-offs, Gotchas, and What Goes Wrong

The sections above describe the XDP load balancer design when it works. This one covers what breaks it.

Figure 4: How eviction plus a ring change turns a healthy flow into a reset.

Connection resets on backend change

Figure 4 is the signature failure. A flow starts and is pinned to Backend A. Later, either an LRU eviction or a balancer restart removes the pinned entry, and in the meantime the ring has changed because Backend B was added or Backend C was drained. The next packet hashes to Backend B, which has no TCP state for this connection and answers with a reset. To the client, an established connection has simply died.

Mitigations are about narrowing the window. Size the connection table above your peak concurrent flows with margin. Add backends gradually, and drain rather than remove, so that the old backend keeps accepting its existing flows while new flows go elsewhere. Avoid restarting balancer nodes during backend changes. Also remember that the Maglev table disrupts a small fraction of unrelated slots on any change, as the simulation showed, so a change to one backend can occasionally affect flows that were assigned elsewhere.

MTU and fragmentation

IPIP adds 20 bytes. The Katran README states that it does not support fragmentation: it cannot forward or fragment oversized packets, and its default maximum packet size is 1.5k with an upper bound of 3.5k. It recommends lowering the TCP MSS advertised by L7 balancers, for example from 1460 to 1450 for IPv4, to avoid generating packets that would not fit after encapsulation. The practical alternative is a larger MTU on the path between the balancer and the backends, such as jumbo frames, which is the approach to prefer if you control the network. Cilium’s option-based DSR has the same problem in another form, because adding to the packet reduces the effective MTU, which is why it offers hybrid mode.

Packets with IP options are also unsupported in Katran, and some networks drop packets carrying options, which is why Cilium offers Geneve and IPIP as alternatives to its option-based dispatch.

Driver support, tooling and observability

Native XDP needs driver support. A fleet with mixed NICs will have some hosts that fall back to generic mode and quietly lose most of the performance. Check the attach mode on every host, not just in the staging environment. The cloud caveats above, with ENA version and MTU limits, are a good example of how specific the conditions can be.

Debugging changes too. Redirected or transmitted packets bypass the stack, so tcpdump on the interface does not see packets the XDP program handled, as the Cilium documentation points out. You need program-level counters, per-CPU statistics maps and tools like bpftool to see what the balancer is doing. Build those dashboards before the first incident.

Topology constraints and verifier limits

Katran assumes a routed network above the top-of-rack switch, sends return-path traffic to the first routing device, and targets a load-balancer-on-a-stick layout where one interface carries both directions. That is easy to satisfy in a modern leaf-spine fabric and awkward in a flat Layer-2 network. DSR also means the balancer never sees replies, so it cannot observe connection teardown, which is part of why it relies on LRU eviction rather than TCP state tracking.

Finally, the verifier limits program complexity, and kernel versions differ in what they allow. Katran’s README lists Linux 5.6 or newer and Clang 6.0 or newer. Features such as multi-buffer packets, which matter for jumbo frames, have evolved across kernel releases; I have not verified which release added support in each driver, so check your kernel and driver documentation before depending on them.

Practical Recommendations

Start by deciding whether you need an XDP load balancer at all. If your edge handles a few hundred thousand packets per second, IPVS or an L7 proxy is simpler and perfectly adequate. The case for XDP appears when the packet rate is high enough that cores spent on forwarding are a visible cost, or when you run Kubernetes and want NodePort and LoadBalancer services without kube-proxy overhead.

If you adopt Katran as your XDP load balancer, treat the ring and the connection table as the two things to protect. Keep configuration identical on every node and push changes atomically. Size the table from your measured concurrent flows. Choose M as a prime at least 100 times your largest backend pool. Keep the MSS or MTU arithmetic written down next to the topology diagram.

If you adopt Cilium, start with best-effort acceleration in a test cluster, confirm XDP Acceleration: Native where you expect it, then move to native once every device is confirmed. Use maglev for external traffic if resilience to node churn matters, set a shared hash seed, and choose a table size that fits your backend count. Prefer IPIP or Geneve DSR over the option-based method unless you control every network element.

A short checklist:

Verify native XDP attach mode on every host and alarm on fallback to generic.
Measure per-core Mpps on your own hardware with realistic packet sizes before sizing the fleet.
Use identical backend ordering, weights and hash seed on all balancer nodes.
Size the LRU table above peak concurrent flows, and monitor evictions.
Drain backends before removing them; serialize node restarts and backend changes.
Budget for the encapsulation overhead in MTU or MSS, and test with full-size packets.
Build program-level metrics, because tcpdump will not show XDP-handled traffic.

Frequently Asked Questions

What is an XDP load balancer?

An XDP load balancer is a Layer-4 forwarder implemented as an eBPF program attached to the network driver’s receive path. It inspects each packet before the kernel allocates a socket buffer, chooses a backend by connection table or consistent hash, encapsulates the packet, and sends it out with XDP_TX. Katran and Cilium’s accelerated NodePort path are examples. Because it skips most of the stack, it needs far fewer cycles per packet than netfilter-based balancing.

Is XDP faster than DPDK?

In raw single-core packet drop, no: the 2018 XDP paper measured 24 Mpps for XDP and 43.5 Mpps for DPDK. For a full forwarding workload, the gap depends on the implementation and core count; the paper reports XDP forwarding surpassing DPDK at two or more cores in one test. XDP’s advantage is operational: it keeps the NIC in the kernel, does not need dedicated polling cores, and works with standard tooling.

How does Katran keep connections stable when backends change?

Katran uses two layers. A fixed-size LRU connection table pins each active flow to the backend that accepted it, so ring changes do not move established connections. For new flows, or flows whose entry was evicted, a Maglev-derived consistent hash over the five-tuple selects the backend, and because the hash uses only packet headers, different Katran nodes choose the same backend without sharing state.

What is Maglev hashing and why must the table size be prime?

Maglev hashing fills a lookup table of M slots by having each backend walk its own permutation of the slots, claiming the first free one in turn. The permutation is (offset + j times skip) mod M, which visits every slot exactly once only if M is prime. The paper recommends M above 100 times the backend count so each backend’s share differs by at most about 1 percent.

Does Cilium use XDP for all Kubernetes service traffic?

No. Cilium uses socket-level load balancing for traffic originating inside the cluster, rewriting the service address at connect time. XDP acceleration applies to external traffic arriving at a node for NodePort, LoadBalancer and externalIPs services, and it must be enabled with loadBalancer.acceleration. Maglev hashing in Cilium also applies only to external north-south traffic, not to east-west connections.

What breaks most often in production?

Connection resets after a backend or ring change combined with lost table entries, and MTU problems caused by encapsulation overhead are the two most common failures. Driver support is a close third: hosts that silently fall back to generic XDP lose much of the benefit. All three are avoidable with a sized connection table, MSS or MTU planning, and an attach-mode check on every host.