io_uring Explained: Linux Async I/O, Rings and Fast Servers

Most Linux network servers still spend a surprising share of their CPU time not on your code, but on crossing the user and kernel boundary. Every recv, send, accept and epoll_wait is a system call, and each one carries fixed costs that grow painful as devices get faster and request sizes shrink. io_uring is the Linux kernel’s answer: two ring buffers shared between your process and the kernel, through which you queue work and collect results with few or no system calls at all.

It matters now because the interface has matured. Multishot operations, provided buffer rings, zero-copy send and a rapidly developing zero-copy receive path make it a credible foundation for storage engines, proxies and databases, while the security conversation around it has become more nuanced. This post explains how the rings actually work, why a completion model differs from epoll readiness, which kernel release introduced each feature, and how to write a liburing server that uses them. You will also see where io_uring is the wrong tool.

What this covers: the syscall-cost problem, the shared ring memory layout and barriers, SQE opcodes and chaining, registered resources, SQPOLL, multishot and buffer rings, zero-copy send, a runnable echo server, security posture, comparisons with epoll, AIO and SPDK, and failure modes.

Context and Background

Linux has had asynchronous I/O interfaces for a long time, and each one left a gap. POSIX select and poll scale linearly with the number of descriptors. epoll, added in the 2.5 series and the backbone of nginx, Node.js and most event loops since, fixed that scaling problem but only tells you that a descriptor is ready. You still issue the actual read or write as a separate system call. Regular files are worse: epoll cannot watch them at all (epoll_ctl fails with EPERM), so event loops push disk work to thread pools.

Linux native AIO (io_submit, io_getevents) tried to offer true asynchronous submission, but it only behaves asynchronously for O_DIRECT files, can block on metadata, and does not cover sockets in practice. Jens Axboe, the block-layer maintainer, wrote about these deficiencies in the original design paper, “Efficient IO with io_uring” (version 0.4 is dated 2019-10-15 and is hosted at kernel.dk). io_uring merged in Linux 5.1 in 2019 and has been growing ever since, to the point that the kernel now treats it as a general asynchronous system-call mechanism rather than a storage API.

The design goal in the paper is explicit: an interface with one producer and one consumer per ring so that no locks are needed, that supports any kind of I/O (buffered, direct, sockets), and that can be driven with as few system calls as possible. The man page for io_uring(7) summarizes the model: you add submission queue entries (SQEs) at the tail of the submission queue (SQ), the kernel consumes them from the head, and the kernel posts completion queue entries (CQEs) that you consume from the CQ head.

If you have read our posts on XDP and eBPF Layer 4 load balancing or on LSM-tree compaction in RocksDB, you have seen two places where this matters. Packet processing in XDP bypasses most of the stack, while storage engines like RocksDB are increasingly exploring async I/O to hide read latency during compaction and point lookups. io_uring sits between those extremes: it keeps the normal kernel stack and its protocol correctness, but reduces the per-operation tax.

One framing note on terminology. “Linux async io” in common speech means three different things: nonblocking sockets with readiness notification, kernel-side asynchronous completion, and thread-pool offload that fakes asynchrony. io_uring is the second kind, and that distinction drives everything below.

The Syscall-Cost Problem and the Ring Architecture

A system call costs more than the instructions in the kernel handler. The CPU transitions privilege levels, the kernel may flush or switch state for speculative-execution mitigations, caches and branch predictors are disturbed, and the instruction stream is interrupted. The exact price depends on the CPU generation and mitigation settings, so treat any single “nanoseconds per syscall” figure you see online as machine-specific rather than universal. What is robust is the structure of the problem: a typical epoll server pays one epoll_wait plus at least one recv and one send per request, so a server doing a million small requests per second performs millions of kernel crossings per second before doing any application work.

Direct answer: io_uring replaces per-operation system calls with two shared-memory ring buffers. The application writes submission queue entries describing the work, the kernel executes it and writes completion queue entries with the results. One io_uring_enter call can submit and reap many operations, and with kernel-side polling it can be avoided entirely.

Figure 1: The io_uring data path. The application fills SQEs and advances the SQ tail, the kernel consumes them, performs the I/O and posts CQEs to the completion ring.

Figure 1 shows the path. The application and kernel share three memory regions created by io_uring_setup(2): the SQ ring (head, tail, mask, flags and an index array), the CQ ring (head, tail, mask and the CQE array itself) and the SQE array. You obtain them with mmap. On kernels 5.4 and later the feature flag IORING_FEAT_SINGLE_MMAP lets the SQ and CQ rings share one mapping, and the man page recommends testing the feature flag rather than the kernel version. liburing handles this for you.

Why the SQ has an indirection array

The original design stores SQE slots in a separate array, and the SQ ring holds indices into it. The CQ, by contrast, indexes its entries directly. The paper explains the reason: indirection lets an application embed SQEs inside its own per-request structures and submit them in any order. A later setup flag, IORING_SETUP_NO_SQARRAY (kernel 6.6 per the setup man page), removes the array so the SQ indexes SQEs directly, which saves a cache line of bookkeeping for the common case.

An SQE is a 64-byte structure containing the opcode, flags, I/O priority, file descriptor, offset, buffer address, length, an operation-specific flags union and a user_data field. A CQE is 16 bytes in the original layout: user_data copied unchanged from the submission, res holding what the equivalent syscall would have returned (or a negative errno), and flags. Larger 128-byte SQEs (IORING_SETUP_SQE128, kernel 5.19) and 32-byte CQEs exist for passthrough commands and for zero-copy receive.

Ring counters are free-running 32-bit integers masked to find slots, so entry counts must be powers of two. The paper gives a range of 1 to 4096 entries for the original design, and the CQ defaults to twice the SQ size. Newer kernels have raised limits, so check the man page of your kernel rather than relying on those figures.

Memory ordering without locks

Both sides write to shared head and tail indices, so the protocol depends on memory ordering rather than mutexes. The man page states the rule: the producer publishes a new tail with a release store, and the consumer reads the peer’s index with an acquire load. For submission, the application fills the SQE, then performs a release store to the SQ tail. The kernel acquires the tail, reads the SQE and later advances the SQ head. For completion the roles reverse.

Figure 3 later in this post walks through one such exchange. The practical advice from both the paper and the man page is the same: do not write these barriers yourself. liburing wraps them (io_uring_smp_store_release and friends), including correct behavior on weakly ordered architectures such as ARM, where a missing barrier can show up as a rare torn read months later.

Completion order and the user_data contract

Requests may complete in any order. The kernel posts one CQE per SQE in the simple case, though flags and multishot operations change that. Because order is not guaranteed, user_data is the only correlation key you have. Typical servers pack a connection pointer or a small tagged integer (connection index plus operation type) into the 64 bits. That convention, an operation type encoded alongside a connection identity, is the backbone of every io_uring server loop you will read.

Inline completion and the async fallback

When you submit a read that hits the page cache, or a send that the socket can accept immediately, the kernel often completes the request inline during io_uring_enter and the CQE is already visible when the call returns. If the operation cannot proceed, io_uring arms internal poll handlers for sockets and pipes, or punts the work to a kernel worker thread (the io-wq pool) for operations that must block, such as buffered file I/O that misses cache. This is a major difference from AIO: you can submit anything and trust the kernel to pick the cheapest correct execution path.

SQE Opcodes, Linking and Ordering

The set of SQE opcodes has grown from reads and writes to a catalog that covers most of what a server does. Core file and socket opcodes include IORING_OP_READV, IORING_OP_WRITEV, IORING_OP_READ, IORING_OP_WRITE, IORING_OP_ACCEPT, IORING_OP_CONNECT, IORING_OP_SEND, IORING_OP_RECV, IORING_OP_SENDMSG, IORING_OP_RECVMSG, IORING_OP_CLOSE, IORING_OP_OPENAT and IORING_OP_STATX. Control opcodes include IORING_OP_NOP, IORING_OP_TIMEOUT, IORING_OP_LINK_TIMEOUT, IORING_OP_ASYNC_CANCEL and IORING_OP_POLL_ADD. Which opcodes exist depends on kernel version, and io_uring_register with IORING_REGISTER_PROBE tells you at runtime.

Linked SQEs

Setting IOSQE_IO_LINK on an SQE chains it to the next: the second request starts only after the first fully succeeds. A failure or short transfer cancels the rest of the chain, with each remaining request completing with -ECANCELED. Chains execute independently of each other. This is how you express “read from this file, then write the result to that socket” or “send, then close” without bouncing through userspace. The cost is subtle: a short read counts as failure for the chain, so you must plan for partial transfers, and for sends the zero-copy man page notes you need MSG_WAITALL for links to treat a short send as an error.

IOSQE_IO_HARDLINK keeps the chain going even on error, and IOSQE_IO_DRAIN holds back the submission queue until all previous requests complete. The paper warns that drain stalls the whole queue, so treat it as a barrier of last resort rather than a normal ordering tool. A link pairs naturally with IORING_OP_LINK_TIMEOUT, which puts a deadline on the previous request and is the cleanest way to implement per-operation timeouts.

Completion semantics you must internalize

Three consequences follow from the CQE model. First, the buffer you hand to recv must stay valid until the CQE arrives, which is a lifetime problem that readiness-based code never had. Second, cancellation is itself an operation: to abandon a pending read you submit IORING_OP_ASYNC_CANCEL, and you must still wait for the original CQE to learn its final status. Third, the CQ can overflow if you submit more than you reap. Kernels with the IORING_FEAT_NODROP feature (5.5) buffer overflowed completions internally and flag the condition, but you should still size the CQ for your worst-case in-flight count.

Registered Resources, SQPOLL and Task Work

The ring removes syscall count as the main cost. What remains are per-operation kernel overheads, and io_uring lets you pay them once instead of on every request.

Fixed files and registered buffers

Every normal I/O request looks up a file descriptor and takes and drops a reference on the underlying file. Registering a file set with IORING_REGISTER_FILES (kernel 5.1) lets you refer to descriptors by index with IOSQE_FIXED_FILE, avoiding that atomic reference traffic. IORING_REGISTER_FILES2 (5.13) added resource tagging and sparse tables, and later “direct descriptors” let accept and open install new descriptors directly into the table without ever creating a normal fd.

Registered buffers, IORING_REGISTER_BUFFERS (5.1), pin memory once so O_DIRECT I/O avoids mapping and unmapping pages on each request. You then use IORING_OP_READ_FIXED and IORING_OP_WRITE_FIXED. The register man page notes these pages are locked and charged against RLIMIT_MEMLOCK, so a non-privileged process with a low limit gets ENOMEM. Sparse buffer tables (IORING_RSRC_REGISTER_SPARSE) arrived in 5.19.

SQPOLL: submission without syscalls

With IORING_SETUP_SQPOLL, a kernel thread polls the SQ ring, so the application can submit simply by writing SQEs and updating the tail. The thread sleeps after an idle timeout (one second by default in the original design, configurable in milliseconds), sets IORING_SQ_NEED_WAKEUP in the ring flags, and the application must then call io_uring_enter with IORING_ENTER_SQ_WAKEUP. You can pin it with IORING_SETUP_SQ_AFF. Unprivileged SQPOLL arrived gradually: kernel 5.11 allowed it with CAP_SYS_NICE, and 5.13 removed the special requirement.

SQPOLL is a throughput-for-CPU trade. You dedicate a core (or part of one) to polling, and it only pays off when submission rates are high enough to keep that thread busy. For a service at modest load, SQPOLL burns CPU for nothing and can lower efficiency per request. The paper also notes that with SQPOLL submitted pointers must stay valid until completion.

Task work, cooperative running and deferred execution

Completions that need to run in the context of the submitting task are queued as task work, and by default the kernel may interrupt your thread (via an IPI) to run it. Three setup flags control that behavior. IORING_SETUP_COOP_TASKRUN (5.19) avoids the interrupt and runs task work when the thread next transitions into the kernel. IORING_SETUP_SINGLE_ISSUER (6.0) promises that only one task submits, enabling cheaper locking. IORING_SETUP_DEFER_TASKRUN (6.1) goes further and defers completion work until the application explicitly asks for events through io_uring_enter with GETEVENTS, which batches work and gives you control over when it happens. These flags are what the high-performance server guides actually recommend, and the combination of SINGLE_ISSUER plus DEFER_TASKRUN is also a prerequisite for zero-copy receive.

io_uring vs epoll: Readiness Versus Completion

The deepest difference between the two is not speed. It is who performs the I/O. With epoll you ask “tell me when this socket can be read without blocking”, then you perform the read. With io_uring you say “read into this buffer and tell me when it is done”. The first is a readiness model and the second is a completion model, the same split that separates Unix reactors from Windows IOCP.

Figure 2: The epoll loop crosses into the kernel at least three times per request. The io_uring loop queues operations in shared memory and crosses once per batch.

Figure 2 contrasts the loops. In the epoll column, a request costs an epoll_wait that discovers readiness, a recv that copies data, and a send for the reply. Edge-triggered mode and batching of epoll_wait results amortize the wait, but recv and send remain one-to-one with requests. In the io_uring column, the application queues accept, recv and send SQEs, calls io_uring_enter once for the whole batch, and later reaps CQEs that already contain the received bytes’ length and buffer.

Where the savings come from

Three sources are real and separable. Fewer kernel crossings is the obvious one, and it grows with batch size: if you handle 64 ready connections per wakeup, you replace roughly 128 syscalls with one. Second, kernel-internal fast paths such as fixed files and registered buffers remove per-request bookkeeping. Third, io_uring can chain dependent operations in the kernel, so “read file, send to socket” never wakes userspace in between.

What does not change is the work of the network stack itself. TCP processing, the skb allocations and the memory copies between kernel and user buffers remain, unless you adopt zero-copy send or receive. This explains a pattern in published comparisons: io_uring wins decisively on storage workloads where the old alternative was blocking threads or AIO, and wins modestly on network workloads where epoll was already efficient.

What the primary sources claim

The design paper includes the author’s own measurements. On the test system, polled 4K random reads reached about 1.7 million IOPS with io_uring versus about 608 thousand for AIO (which did not support polling in that test), and non-polled 4K random reads reached about 1.2 million IOPS, roughly twice AIO. The paper reports peak per-core 4K IOPS of about 1,700K and no-op message rates of 12 to 20 million per second depending on the machine. These are single-author results from 2019 hardware, the paper itself marks them as partly outdated, and they measure the interface, not an application. Use them as evidence of direction, never as a prediction for your workload.

For networking, I am deliberately not quoting any “X percent faster than epoll” number. Public benchmarks of echo servers and proxies vary from near parity to large gains depending on payload size, kernel version, mitigation settings and whether the epoll baseline was well tuned. The honest summary is that the advantage scales with how many syscalls your epoll design performs per unit of useful work, which is highest for small messages and many connections. Measure on your own hardware with your own traffic.

A decision matrix

Workload	epoll	Linux AIO	io_uring
Many idle sockets, little traffic	Excellent, simple	Not applicable	Fine, no gain
Small messages, high connection counts	Good, syscall bound	Not applicable	Best, fewer crossings
Buffered file reads	Needs thread pool	Blocks silently	Async, inline when cached
O_DIRECT NVMe at high IOPS	Not applicable	Works, limited	Best, with polling
Kernel older than 5.10 or locked-down sandbox	Safe default	Available	Risky or unavailable
Portability beyond Linux	kqueue analogue	None	None

Multishot, Provided Buffers and the Modern Server Loop

Early io_uring servers were disappointing in one respect: you had to resubmit an accept after every connection and a recv after every message. That consumed SQEs and reintroduced work per operation. Two later features address this and together define what a “modern” io_uring network server looks like.

Multishot operations

A multishot request stays armed and produces many CQEs from one SQE. Multishot accept arrived in kernel 5.19 according to my notes on the release, which I could not confirm against the primary man page this run, so verify against your target kernel. Multishot receive is documented in the liburing man page for io_uring_prep_recv_multishot as available since kernel 6.0. When a multishot request is still armed, each CQE carries IORING_CQE_F_MORE. When that flag is absent, the request has terminated (error, cancellation or buffer exhaustion) and you must re-arm it.

Provided buffer rings

Receiving without knowing which connection will have data next is the problem. If every pending recv needs its own buffer, a server with 100,000 idle connections pins 100,000 buffers. Provided buffers solve this by making the kernel select a buffer from a pool at completion time: you set IOSQE_BUFFER_SELECT and a buffer group id, and the CQE reports which buffer was chosen via the buffer id in its flags. The man page states that multishot receive requires IOSQE_BUFFER_SELECT.

The ring-mapped variant, IORING_REGISTER_PBUF_RING (5.19 per the register man page), places buffer descriptors in a shared ring so the application can replenish buffers without a syscall, using the same single-producer pattern as the SQ. The older IORING_OP_PROVIDE_BUFFERS opcode exists for earlier kernels, at the cost of one SQE per batch. After processing a buffer you hand it back to the ring; if the pool runs dry the kernel terminates the multishot receive with -ENOBUFS (a behavior I know from the kernel and liburing sources and did not find in the man page fetched this run), and the application must replenish and re-arm.

Figure 4: A per-core server loop using multishot accept, multishot recv with a provided buffer ring, and zero-copy send with notification CQEs.

Figure 4 shows the lifecycle. A listener (ideally one ring per core, with SO_REUSEPORT spreading connections) arms a multishot accept. Each accepted connection gets a multishot recv drawing from a buffer group. Completed requests produce a response, sent either with a normal send or SEND_ZC, and the buffer returns to the group. A CQE without the MORE flag triggers re-arming.

Zero-copy send

IORING_OP_SEND_ZC avoids copying user data into socket buffers: the NIC (or the stack, for loopback and some paths) reads directly from your pages. The liburing manual for io_uring_prep_send_zc documents the contract. A zero-copy send usually generates two CQEs. The first carries the send result and, if IORING_CQE_F_MORE is set, a second notification CQE with IORING_CQE_F_NOTIF follows when the memory is safe to reuse. Until the notification arrives, you may not modify or free the buffer. I recall SEND_ZC landing in kernel 6.0; the man page I fetched dated September 2022 did not print a version, so treat 6.0 as unverified.

Zero-copy is not free. Pinning pages and tracking notifications has overhead, so it tends to pay off for large payloads (the kernel documentation and community guidance point to tens of kilobytes and above, but the exact crossover is workload specific and I am not citing a number). The flag IORING_SEND_ZC_REPORT_USAGE tells you in the notification whether the kernel fell back to copying, which is the right diagnostic when zero-copy appears to give no gain. The fixed variant requires buffers registered with io_uring_register_buffers.

Zero-copy receive

The receive side is harder, because the NIC must deliver payloads into memory the kernel hands to userspace. The kernel documentation for io_uring zero-copy Rx describes the setup: headers go through the normal TCP stack into kernel memory while payloads land in userspace buffers. The NIC must support header/data split, flow steering and RSS, configured with ethtool, with queues reserved for zero-copy flows (the page asks for at least two queues, half reserved via RSS). The ring must be created with IORING_SETUP_SINGLE_ISSUER, IORING_SETUP_DEFER_TASKRUN and either IORING_SETUP_CQE32 or IORING_SETUP_CQE_MIXED. The fetched page does not state the introduction version (my recollection is a 6.15 merge; unverified), and driver support is limited, so this is for teams with control over their NICs.

Walk-through: A liburing Echo Server

The following program is a minimal single-threaded echo server using liburing. It demonstrates the submission and completion cycle, user_data tagging and the CQE loop, with multishot accept and a plain per-connection buffer for clarity. Compile with gcc -O2 echo.c -o echo -luring. It requires a kernel with multishot accept (I believe 5.19 or newer) and liburing development headers.

#include <liburing.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#define PORT 9000
#define QD 256
#define BUFSZ 2048

enum { OP_ACCEPT, OP_RECV, OP_SEND };

struct conn { int fd; int op; char buf[BUFSZ]; };

static inline uint64_t tag(struct conn *c, int op) {
    return (uint64_t)(uintptr_t)c | (uint64_t)op;   /* ptr aligned to 8 */
}

int main(void) {
    int lfd = socket(AF_INET, SOCK_STREAM, 0);
    int one = 1;
    setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
    struct sockaddr_in a = { .sin_family = AF_INET,
        .sin_port = htons(PORT), .sin_addr.s_addr = INADDR_ANY };
    if (bind(lfd, (void *)&a, sizeof(a)) || listen(lfd, 1024)) {
        perror("bind/listen"); return 1;
    }

    struct io_uring ring;
    struct io_uring_params p = {0};
    p.flags = IORING_SETUP_COOP_TASKRUN | IORING_SETUP_SINGLE_ISSUER;
    if (io_uring_queue_init_params(QD, &ring, &p) < 0) {
        perror("queue_init"); return 1;
    }

    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
    io_uring_prep_multishot_accept(sqe, lfd, NULL, NULL, 0);
    io_uring_sqe_set_data64(sqe, OP_ACCEPT);   /* no conn yet */

    for (;;) {
        io_uring_submit_and_wait(&ring, 1);
        struct io_uring_cqe *cqe;
        unsigned head, n = 0;
        io_uring_for_each_cqe(&ring, head, cqe) {
            n++;
            uint64_t ud = io_uring_cqe_get_data64(cqe);
            int op = (int)(ud & 7);
            struct conn *c = (struct conn *)(uintptr_t)(ud & ~7ULL);

            if (op == OP_ACCEPT) {
                if (cqe->res >= 0) {
                    struct conn *nc = aligned_alloc(8, sizeof(*nc));
                    nc->fd = cqe->res;
                    sqe = io_uring_get_sqe(&ring);
                    io_uring_prep_recv(sqe, nc->fd, nc->buf, BUFSZ, 0);
                    io_uring_sqe_set_data64(sqe, tag(nc, OP_RECV));
                }
                if (!(cqe->flags & IORING_CQE_F_MORE)) {  /* re-arm */
                    sqe = io_uring_get_sqe(&ring);
                    io_uring_prep_multishot_accept(sqe, lfd, NULL, NULL, 0);
                    io_uring_sqe_set_data64(sqe, OP_ACCEPT);
                }
            } else if (op == OP_RECV) {
                if (cqe->res <= 0) { close(c->fd); free(c); continue; }
                sqe = io_uring_get_sqe(&ring);
                io_uring_prep_send(sqe, c->fd, c->buf, cqe->res, 0);
                io_uring_sqe_set_data64(sqe, tag(c, OP_SEND));
            } else { /* OP_SEND done, read again */
                if (cqe->res < 0) { close(c->fd); free(c); continue; }
                sqe = io_uring_get_sqe(&ring);
                io_uring_prep_recv(sqe, c->fd, c->buf, BUFSZ, 0);
                io_uring_sqe_set_data64(sqe, tag(c, OP_RECV));
            }
        }
        io_uring_cq_advance(&ring, n);
    }
}

Several details in this code are the heart of correct io_uring programming. The tag packs an operation code into the low three bits of an 8-byte-aligned pointer, which is why aligned_alloc(8, ...) is used. The connection buffer stays alive until its CQE, satisfying the lifetime rule from earlier. io_uring_for_each_cqe plus a single io_uring_cq_advance reaps a batch with one release store to the CQ head. The code is a teaching sketch and omits partial-send handling (if res is less than requested the remainder must be resent), SQ-full handling (io_uring_get_sqe returns NULL when the SQ is full, in which case you must submit and retry), error classification and graceful shutdown.

Figure 3: One round trip. The release store on the SQ tail publishes the SQE, the kernel’s release store on the CQ tail publishes the CQE, and each reader pairs it with an acquire load.

Figure 3 shows what each liburing call is doing underneath. io_uring_submit ends in io_uring_enter (or skips it under SQPOLL if the thread is awake), while io_uring_cq_advance is the consumer’s release store of the CQ head, returning slots to the kernel.

Upgrading the sketch into a production loop

To move from sketch to server, make five changes. Replace per-connection buffers with a provided buffer ring and multishot recv so idle connections hold no memory. Use fixed files by accepting into direct descriptors. Add SEND_ZC for large responses behind a size threshold. Run one ring per core, each with its own listener under SO_REUSEPORT, so rings never share locks. And put DEFER_TASKRUN on the ring and reap with io_uring_submit_and_wait, so task work executes at a time you control. A short sketch of the buffer-ring setup follows.

struct io_uring_buf_ring *br;
int ret;
br = io_uring_setup_buf_ring(&ring, NBUFS, BGID, 0, &ret);
for (int i = 0; i < NBUFS; i++)
    io_uring_buf_ring_add(br, pool + i * BUFSZ, BUFSZ, i,
                          io_uring_buf_ring_mask(NBUFS), i);
io_uring_buf_ring_advance(br, NBUFS);

sqe = io_uring_get_sqe(&ring);
io_uring_prep_recv_multishot(sqe, fd, NULL, 0, 0);
sqe->flags |= IOSQE_BUFFER_SELECT;
sqe->buf_group = BGID;

On each CQE with IORING_CQE_F_BUFFER set, cqe->flags >> IORING_CQE_BUFFER_SHIFT gives the buffer id. After handling it, return the buffer with another io_uring_buf_ring_add and io_uring_buf_ring_advance. Confirm the function signatures against your installed liburing version, since the helper APIs have been refined across releases.

Security Posture, Sandboxes and Where io_uring Is Disabled

The same property that makes io_uring fast, a large in-kernel attack surface reachable through a single entry point, makes it controversial. Because requests execute inside the kernel with their own opcodes, an io_uring call can perform operations that a seccomp filter on open, connect or read never sees, since those syscalls are never made. Security tooling that hooks syscalls can be blind to I/O issued through the ring, a point researchers demonstrated in 2025 with a proof-of-concept rootkit that did its work entirely through io_uring.

What Google and Docker did

In June 2023 Google reported that io_uring accounted for 60 percent of the submissions to its Vulnerability Rewards Program, as reported by Phoronix. In response Google disabled io_uring on ChromeOS until it could be sandboxed properly, blocked apps from using it on Android with a seccomp-bpf filter (with a plan to restrict it further through SELinux), was working to disable it by default in GKE Autopilot, and disabled it on its production servers. Docker subsequently removed io_uring syscalls from its default seccomp profile, which is why a container that works on your laptop may fail with ENOSYS or EPERM on a hardened runtime. I did not verify the specific Docker or containerd release numbers, so check your runtime’s default profile directly.

The counter-argument

Defenders point out that the vulnerabilities were bugs in implementation, found at a high rate precisely because researchers were rewarded for looking, and that the code has been hardened since. The Wikipedia summary of a 2026 talk by Jens Axboe quotes him disputing that io_uring still has major security problems, calling it “no worse, I would argue even better” than epoll or futex. I could not verify the talk firsthand, so treat it as the maintainer’s position rather than an audit result. Whether you agree, the kernel has added controls: IORING_REGISTER_RESTRICTIONS (5.10) lets the creator of a ring limit the opcodes and flags it will accept before enabling it, a sysctl (kernel.io_uring_disabled, which I recall from kernel documentation and did not re-verify this run) can disable creation globally or for unprivileged users, and per the same Wikipedia page Linux 7.0 added BPF-based submission queue filtering comparable to seccomp-bpf.

A practical stance

Treat io_uring like any powerful interface. If you run untrusted code, multi-tenant containers or a hostile-input environment, leave it blocked unless you have a clear need, and if you enable it, do so in a dedicated service with restrictions registered at ring creation. If you control the whole machine, as in a database appliance or a single-tenant storage node, the risk calculus is much closer to using any other privileged kernel feature. Record the decision and the kernel version in your threat model, because the answer changes with patch level.

How It Compares: AIO, SPDK, Thread Pools and Runtimes

io_uring versus Linux AIO

The AIO interface works asynchronously only for O_DIRECT and has no networking story, so for new code there is little reason to choose it. Databases that already depend on libaio can keep it. Migration to io_uring typically gets you buffered I/O support and a smaller syscall footprint, but changes your buffer lifetime and error handling code.

io_uring versus SPDK

The Storage Performance Development Kit takes the opposite approach: it moves the NVMe driver into user space, polls device queues directly, and bypasses the kernel entirely. That removes kernel overhead completely but also removes the filesystem, page cache, permissions model and the kernel’s scheduling. SPDK suits dedicated storage appliances that own the device and want the highest possible IOPS per core. io_uring with IORING_SETUP_IOPOLL keeps the kernel’s filesystems and block layer in the loop and is far easier to operate. The paper’s polled-read figures show kernel-based polling can reach the same order of magnitude as dedicated stacks on a single device, but I have no primary-source head-to-head to cite, so test your own device.

Rust and other runtimes

Mainstream tokio is built on epoll through mio and does not use io_uring by default. Separate projects expose the completion model: tokio-uring is a runtime that runs on io_uring, monoio is a thread-per-core runtime that uses io_uring where available, and glommio is another thread-per-core design built around it. I am citing these names from general knowledge without having checked their current maintenance state this run, so verify activity before adopting one. The structural issue they share is the buffer lifetime rule. Rust’s AsyncRead trait hands the kernel a borrowed buffer, which is unsound if a future is dropped while the kernel still writes to it, so these runtimes use ownership-passing APIs where the buffer is moved into the call and returned on completion. Go, Java and Node.js runtimes continue to use epoll internally; the benefits of io_uring there depend on library-level adoption, which is uneven.

Choosing

Choose epoll when your workload is mostly idle connections, you need portability to non-Linux systems, or your kernel is old or locked down. Choose io_uring when you are syscall-bound on small messages, doing buffered or direct file I/O alongside networking, or building a storage engine. Choose SPDK only when you own the device and can accept losing the kernel storage stack.

Trade-offs, Gotchas, and What Goes Wrong

Buffer lifetime bugs. The most common production failure is freeing or reusing a buffer while an in-flight request still references it, particularly on error paths and during connection teardown. Close does not cancel pending operations on a descriptor you registered as a fixed file; you must cancel and then wait for every outstanding CQE before releasing resources.

Partial results. A recv returning fewer bytes than expected is normal, and so is a short send. Linked chains treat short transfers as failures, which surprises people who assume only negative results fail. Always handle res smaller than the requested length.

CQ overflow and backpressure. If completions accumulate faster than you reap them, the CQ overflows. With IORING_FEAT_NODROP the kernel stores the overflow internally, and submission can fail with -EBUSY until you drain. Size the CQ for in-flight operations and reap before you submit more.

Memory limits. Registered buffers and, on kernels 5.11 and earlier, the rings themselves are charged against RLIMIT_MEMLOCK, according to the liburing README and register man page. Container environments with tiny defaults produce ENOMEM that looks unrelated to memory.

Kernel version skew. Opcodes and flags differ across versions, and the liburing project says its regression suite is not expected to pass on older kernels. Use probing (IORING_REGISTER_PROBE) rather than version checks, and keep a fallback path. Distribution kernels backport selectively, so the version string alone misleads.

Debugging and observability. A stuck request looks like a hung process with no blocked syscall to inspect. You need tracing (tracepoints for io_uring exist in the kernel) and your own per-operation timers. Link timeouts are the pragmatic safety net.

SQPOLL waste. A polling thread with nothing to do consumes a core. Enable it only after measuring that the submission rate justifies it.

Fairness and head-of-line effects. A single ring per process can let one chatty connection monopolize the loop. Per-core rings, bounded work per iteration and sensible buffer pool sizes help.

Hype risk. The honest finding of most careful comparisons is that io_uring helps most when syscall overhead is a measurable fraction of your profile. Profile with perf or bpftrace first; if system call time is under a few percent of CPU, the rewrite is unlikely to pay off.

Practical Recommendations

Start by establishing that you have the problem. Profile your current server, and attribute CPU to syscall entry and exit, network stack and application code. If kernel crossings are not a noticeable share, improve something else first.

Adopt incrementally. A pragmatic path is to start with file I/O, where the old alternative was a thread pool, move to socket I/O on a single ring behind an abstraction that can still fall back to epoll, and add advanced features only after measurement. Use liburing rather than raw syscalls so barriers and mmap handling are correct. Prefer multishot receive with provided buffer rings, COOP_TASKRUN or DEFER_TASKRUN with SINGLE_ISSUER, and one ring per core. Keep SQPOLL and zero-copy as measured optimizations rather than defaults.

Decide your security stance up front, and write down which environments will run the new code path. Pin a minimum kernel version for production, test on it in CI, and probe opcodes at startup.

Profile first: confirm syscall overhead is material.
Use liburing, not hand-written barriers.
Tag user_data with operation type and connection identity.
Never free a buffer until its CQE (and any notification CQE) has arrived.
Handle short reads, short writes and missing IORING_CQE_F_MORE.
Use one ring per core and avoid sharing rings between threads.
Add link timeouts to every request that could stall.
Keep an epoll fallback for old kernels and sandboxed environments.
Verify seccomp, container and LSM policy before rollout.

Frequently Asked Questions

Is io_uring faster than epoll?

Often, but not always. io_uring reduces the number of system calls, so it tends to win when requests are small and numerous, and when file and network I/O are mixed. For mostly idle connections or large payloads dominated by TCP processing, the gap narrows to near parity. The primary design paper shows large gains over Linux AIO for storage, but you should benchmark your own network workload on your target kernel before committing.

What is the submission queue in io_uring?

The submission queue is a ring buffer in memory shared with the kernel into which the application writes submission queue entries, each a 64-byte description of an operation such as read, accept or send. The application advances the tail to publish entries and the kernel consumes them from the head. Completed work appears in the separate completion queue, matched back to requests through the user_data field.

Which Linux kernel version do I need for io_uring?

The base interface merged in Linux 5.1. Useful features arrived later: the register man page lists sparse file tables at 5.13, ring-mapped provided buffers at 5.19 and ring fd registration at 5.18, while multishot receive is documented as available since 6.0 and DEFER_TASKRUN since 6.1. Distribution kernels backport selectively, so probe opcodes at runtime and aim for a recent LTS kernel.

Is io_uring safe to use in containers?

It depends on policy. Google restricted it across ChromeOS, Android apps and its production servers after a 2023 report of a high share of bounty submissions, and Docker removed it from its default seccomp profile. You can enable it explicitly, restrict opcodes at ring creation, or leave it blocked for untrusted workloads. Check your runtime’s seccomp profile and kernel sysctls rather than assuming either way.

Does io_uring support zero-copy networking?

Yes, in both directions with caveats. IORING_OP_SEND_ZC sends from user pages and posts a second notification CQE when the buffer may be reused, and it helps most for large payloads. Zero-copy receive is documented in the kernel docs but needs NICs with header/data split, flow steering and RSS, plus rings created with SINGLE_ISSUER, DEFER_TASKRUN and 32-byte CQEs.

Should I rewrite my server with io_uring?

Only if profiling shows syscall overhead or thread-pool file I/O is a real bottleneck. Many services are bound by application logic, locks or the network stack and gain little. If you do adopt it, start with a contained component such as a storage path, wrap it behind an interface, keep an epoll fallback, and settle your security stance first.