CXL Memory Expansion, Pooling and Tiering: Type 3 Devices, Linux NUMA and Workload Fit (2026)

Your servers run out of memory long before they run out of cores, yet adding DRAM means buying a new socket’s worth of DIMM slots, a bigger CPU, or a whole new platform. CXL memory attacks that mismatch by letting a server attach extra DRAM over the same physical lanes used by PCIe, while the operating system still sees ordinary byte-addressable, cache-coherent memory. The catch is that this memory is slower than the DIMMs next to the CPU, so the real engineering question is not “can I attach it” but “which pages should live there.”

This matters now because the specification has moved well beyond the first expansion cards: CXL 4.0 was announced in November 2025, Linux has a mature tiering stack, and cloud operators have published how much DRAM they strand. This post walks the protocol stack, the device types, the three deployment patterns (expansion, pooling, sharing), the Linux plumbing from daxctl to NUMA balancing, latency and cost arithmetic you can reuse, and the workloads where it pays off or quietly hurts.

What this covers: CXL protocols and device types, version differences from 1.1 to 4.0, Linux memory tiering, worked latency and cost models, workload fit for databases, vector search and KV cache offload, and the failure modes to plan for.

Context and Background

Server memory has three problems that compound. DRAM is a large share of server cost; Microsoft’s Azure team has stated that DRAM can be up to roughly half of server cost, as reported by SemiAnalysis summarizing the Pond paper. Capacity per socket is capped by the number of memory channels and DIMMs per channel. And capacity is stranded: when a host has all its cores rented out but unused DRAM left over, nobody else can use that DRAM.

The classic answers each have drawbacks. Buying higher-density DIMMs raises cost per gigabyte steeply at the top of the density curve. Adding sockets adds cores you may not need and brings NUMA penalties. Remote memory over RDMA needs application changes and software paging. Persistent memory in the form of Intel Optane offered a middle tier, but the product line was wound down, which left the tiering software without hardware to manage.

Compute Express Link, or CXL, is the industry answer. It is an open standard from the CXL Consortium that rides on the PCIe physical layer and adds three protocols for coherent access. CXL 1.0 was released in March 2019 on PCIe 5.0, and CXL 1.1 followed in June 2019, per the CXL history summarized on Wikipedia. Intel Sapphire Rapids and AMD Zen 4 EPYC were the first mainstream CPUs announced with CXL 1.1 support.

If you have read our analysis of the memory wall, optical interconnect and LPDDR6, CXL is the capacity-side complement to the bandwidth-side story told there. HBM and LPDDR solve bandwidth per watt close to the accelerator; CXL solves capacity per dollar further out, accepting higher latency in exchange.

Two framing points keep the rest of the article honest. First, CXL memory is not a faster DRAM; it is a different tier with its own latency, bandwidth and failure domain. Second, the value is almost entirely in software placement. A CXL card dropped into a server without tiering policy is an expensive, slow NUMA node that the allocator may fill with your hottest pages by accident.

What problem each CXL generation targets

CXL 1.x is about attaching a device to one host. CXL 2.0 (November 2020, still on PCIe 5.0 signalling) adds single-level switching so multiple hosts can connect to a pool of devices, along with device integrity and data encryption. CXL 3.0 (August 2022) moves to PCIe 6.0 at 64 GT/s, adds multi-level switching, memory sharing and peer-to-peer access. CXL 3.1 (November 2023) and 3.2 (December 2024) are incremental. CXL 4.0 (November 2025) doubles the signalling rate to 128 GT/s. We cover each in the architecture section.

How CXL Memory Works: Protocols, Device Types and Topologies

The short version: CXL carries three protocols over one PCIe-style link. CXL.io handles discovery and configuration, CXL.cache lets a device coherently cache host memory, and CXL.mem lets the host load and store to device-attached memory. A Type 3 device uses CXL.io plus CXL.mem, which is exactly what a memory expander is.

Figure 1: Host root port, the three CXL protocols and the three device types. A Type 3 device (CXL.io plus CXL.mem) is the memory expander.

Figure 1 shows the split. The host CPU has its normal memory controller driving local DDR5, plus one or more root ports that speak CXL over PCIe lanes. Behind a root port, CXL.io is mandatory for every device because it is based on PCIe and handles configuration, device discovery, interrupts and DMA. CXL.cache and CXL.mem are optional layers on top. The device types are defined by which of those layers they implement.

The three device types

Type 1 devices implement CXL.io and CXL.cache. They have no local memory exposed to the host; they are accelerators such as smart NICs that want coherent access to host memory so they can cache it without software-managed DMA. Type 2 devices implement all three protocols and carry their own memory (GDDR or HBM), so both sides can cache each other’s data. Before CXL 3.0, Type 2 devices managed coherence with device-bias and host-bias modes set per 4 KB page; CXL 3.0 replaced bias with enhanced coherency, including back-invalidation of host caches.

Type 3 devices implement CXL.io and CXL.mem. The host treats the device’s memory as an extension of its address space using ordinary loads and stores. The device does not cache host memory and does not initiate coherent requests, which is why it is the simplest type to build and the first to reach volume. A Type 3 device can be an add-in card, an E3.S drive-form-factor module, or a memory buffer on a system board.

Why the latency is higher than a DIMM

A CXL.mem access from a core goes through the CPU’s cache hierarchy, out the root port, across the PCIe-style serializer, through the device’s CXL controller, to a DDR5 controller on the device, and back. Every hop adds delay. The Wikipedia summary of the standard puts the typical added latency of a CXL memory controller at about 200 ns, though that is a general characterization rather than a measurement of one product.

Research gives a more careful decomposition. The Pond paper (Microsoft, ASPLOS 2023, arXiv 2203.00241) reports an Intel-measured CXL port round trip of 25 ns and estimates the total CXL overhead at roughly 70 to 90 ns for pools of 8 to 16 sockets, rising above 180 ns for rack-scale pools that need switches and retimers. Those are modeled and emulated figures, because the paper predates shipping hardware. Real devices vary by controller design, DDR speed grade and link width, so treat any single number as a range, not a constant.

A useful mental model for planning: local DRAM is roughly 80 to 120 ns; a direct-attached CXL expander lands in roughly the range of one extra NUMA hop to about 2.5 times local DRAM latency; a switched pool sits above that. The Pond paper’s own emulation used 182% and 222% of local latency as its two operating points, based on a 142 ns remote-socket latency on an Intel prototype (78 ns local) and 255 ns on an AMD prototype (115 ns local). Those ratios, not absolute nanoseconds, are what you should carry between hardware generations.

Bandwidth: a x16 link is not a DDR channel

Bandwidth is a separate axis. On PCIe 5.0 signalling at 32 GT/s, a x16 link carries about 63 GB/s per direction before protocol overhead. A single DDR5-4800 channel peaks at 38.4 GB/s, and DDR5-5600 at 44.8 GB/s, both computed from transfer rate times 8 bytes. So one x16 CXL link has the raw capacity of roughly 1.4 to 1.6 DDR5 channels, and CXL uses a link that is full-duplex, so the read/write mix matters: a pure read stream cannot use the write direction.

In practice, usable bandwidth is below the raw figure due to flit overhead and controller efficiency. Many expander cards use x8 links (about 31.5 GB/s raw per direction at PCIe 5.0), which is less than a single DDR5-4800 channel. That is why CXL memory is better described as capacity expansion with moderate bandwidth, not a way to add bandwidth. CXL 3.0’s move to 64 GT/s (121 GB/s per direction at x16 raw) and CXL 4.0’s 128 GT/s (about 242 GB/s at x16 raw, per the same Wikipedia table) change that over time.

The words are often blurred, so define them precisely.

Memory expansion attaches a Type 3 device to one host. The capacity belongs to that host only. This is CXL 1.1 and needs no switch.
Memory pooling places memory devices behind a switch and assigns portions to different hosts, one host at a time. A fabric manager binds capacity to hosts and can rebind it later. Each region has exactly one owner at any moment, so no cross-host coherence is needed. This is the CXL 2.0 use case.
Memory sharing lets multiple hosts map the same region simultaneously with hardware coherence between them. This arrived with CXL 3.0 and is the hardest to build and use, because software must coordinate access.

Figure 2: Pooling versus sharing. A fabric manager binds logical devices of a multi-logical-device memory to individual hosts, while a shared region uses CXL 3.x coherence between hosts.

Figure 2 sketches the topology. Hosts connect to a switch. Memory devices behind the switch can present multiple logical devices (a multi-logical-device, or MLD), each of which is bound to one host by the fabric manager, an out-of-band control entity. Pooled capacity is dedicated; shared regions use the coherence machinery added in version 3.

CXL 3.x and 4.0: what changed

CXL 3.0 doubled the link rate to 64 GT/s using PAM-4 signalling on PCIe 6.0, introduced a 256-byte flit (earlier versions used a 68-byte flit), and added multi-level switching, memory sharing, peer-to-peer DMA, and port-based routing that can address up to 4,096 nodes. A latency-optimized flit variant trims roughly 5 ns on the happy path, per the Wikipedia summary. The 3.1 and 3.2 releases refined the fabric and management features; I have not verified their feature lists against the primary specification, so check the CXL Consortium downloads before relying on a specific item.

CXL 4.0, announced on 18 November 2025 per the CXL Consortium press release, doubles bandwidth from 64 to 128 GT/s with no added latency, keeps the 256B flit, adds a native x2 width, supports up to four retimers, introduces bundled ports for Type 1 and Type 2 device ports, and enhances memory RAS. It is backward compatible with 3.x, 2.0, 1.1 and 1.0. As of this writing, I would not expect broad 4.0 silicon in production fleets; the platforms you can buy and deploy for expansion today are CXL 1.1 and 2.0 class.

Where the memory shows up to software

Once the link trains, the platform firmware and the Linux CXL driver stack enumerate the device. Capacity is grouped into a CXL region and exposed in one of two ways. It can appear as a DAX device (a raw, direct-access character device) or it can be hot-plugged into the kernel as a memory-only NUMA node with no CPUs. The second is what most deployments want, because the kernel can then place pages there, and tiering policy can treat it as slower memory.

The firmware also publishes performance attributes. The ACPI HMAT table and the device’s CDAT data describe latency and bandwidth, and the Linux kernel exposes them under /sys/devices/system/node/nodeY/access0/initiators/ as read_latency, write_latency, read_bandwidth and write_bandwidth. The kernel NUMA performance documentation states these correspond to the rated figures for the platform. Rated is not measured: verify with a latency tool before building policy on them.

Making CXL Memory Useful on Linux: Tiering, NUMA and DAX

The short version: the kernel treats CXL memory as a lower-ranked memory tier. Cold pages are demoted from local DRAM to the CXL node under pressure, and hot pages on the CXL node are promoted back when the kernel detects frequent access. You steer this with a handful of sysfs and sysctl knobs, and you can bypass it with explicit NUMA binding when an application knows better.

Figure 3: Demotion on reclaim, promotion on access detection. Tier rank comes from firmware performance data and driver hints.

From NUMA node to memory tier

Historically, a Linux NUMA node with CPUs was the fast tier and a node without CPUs was assumed slower. That implicit rule breaks with new hardware: HBM nodes should be faster, CXL DRAM should sit below local DRAM, and CPU hotplug should not reshuffle tiers. A patch series by Aneesh Kumar and collaborators (discussed on LWN) made tiers explicit. Every node belongs to a tier; the default tier has rank 200; drivers can move their nodes above or below it; and sysfs exposes /sys/devices/system/memtier/ with a memtierN/rank file per tier and /sys/devices/system/node/nodeN/memtier per node. That LWN article is about an RFC revision, so the exact interface details may have evolved before merge; check your kernel’s documentation. The memory-tiers code is present in current mainline, and drivers such as the CXL region code compute an abstract distance used to place devices in tiers.

Demotion and promotion

Demotion is the simpler half. When local DRAM is under pressure, reclaim can migrate cold pages to the next tier down instead of swapping them out. This is controlled by the vm.demotion_enabled setting, exposed in sysfs at /sys/kernel/mm/numa/demotion_enabled; I did not find it in the main sysctl documentation page, so verify it on your target kernel.

Promotion is harder because the kernel must decide a page on the slow tier is hot. Linux reuses automatic NUMA balancing for this: it periodically unmaps pages so that accesses raise hint faults, then migrates frequently faulting pages. The kernel sysctl documentation describes kernel.numa_balancing values 0 (off), 1 (normal balancing toward the accessing node) and 2 (NUMA_BALANCING_MEMORY_TIERING, which places hot pages in faster memory across memory types). It also documents numa_balancing_promote_rate_limit_MBps, which caps promotion and demotion throughput and warns that too high a rate may hurt application latency.

Meta’s TPP (Transparent Page Placement) work shaped this design. According to the TPP paper, it came within 1% of an ideal all-local baseline on Meta’s production-style workloads using early CXL 1.1 CPUs, about 18% better than stock Linux and 5 to 17% better than NUMA Balancing and AutoTiering, and most of its patches landed in Linux 5.18. Those results were measured on Meta’s workloads and prototype hardware; your mileage depends on how skewed your page-access distribution is.

Walk-through: from card to NUMA node

Here is a minimal bring-up using the cxl and daxctl tools. Output formats vary by version, so treat this as a sketch rather than a script.

# 1. Inspect enumerated CXL devices and memdevs
cxl list -M -u

# 2. Create a region from a memdev (interleave over one device)
cxl create-region -m -d decoder0.0 -w 1 mem0

# 3. The region appears as a DAX device; hot-plug it as system RAM
daxctl list -u
daxctl reconfigure-device --mode=system-ram dax0.0

# 4. Confirm a new CPU-less NUMA node and its performance attributes
numactl -H
cat /sys/devices/system/node/node2/access0/initiators/read_latency

# 5. Enable tiering behaviour
echo 1 | sudo tee /sys/kernel/mm/numa/demotion_enabled
sudo sysctl kernel.numa_balancing=2

The daxctl reconfigure-device documentation notes that converting to system-ram hot-plugs the range as regular memory, that the operation is destructive to any data on the device, and that by default memory is onlined with the online_movable policy so that kernel allocations avoid it (daxctl man page). That default is exactly what you want for a slow tier: kernel structures such as page tables and slab stay in local DRAM, while user pages may land on CXL. If you want a different zone policy, use --no-online and manage onlining yourself.

Explicit placement when you know better

Automatic tiering is a heuristic. Databases, caches and inference runtimes often know their own access patterns. numactl --membind and --preferred, set_mempolicy, and mbind let you pin an allocation to the CXL node. The man page for daxctl itself shows numactl --cpunodebind=0-1 --membind=2 -- some-service as the pattern for a service that should live entirely on the converted node.

Explicit binding has a price. If you bind a latency-sensitive component to the CXL node you get the slow tier’s latency for every miss. If you bind a large cold structure there and leave the hot structures free, you get most of the capacity benefit at minimal cost. In practice the best policies are mixed: let the kernel keep the working set in DRAM, and explicitly place large, rarely touched buffers such as caches of cold keys, spilled hash partitions, or old KV cache blocks on the CXL node. The weighted interleave policy added in recent kernels (verify availability on your version) is another option when bandwidth, not capacity, is the goal, because it spreads pages across nodes in a ratio you set.

Worked Latency and Cost Arithmetic

Everything above is qualitative. Planning needs numbers, so here are two small models. All inputs are illustrative assumptions, not measurements; replace them with your own measured values.

Effective latency under tiering

Let local DRAM latency be 100 ns and CXL latency be 250 ns, a 2.5x ratio inside the range the Pond paper explored (1.8x to 2.2x emulated, with larger ratios at rack scale). If a fraction h of memory accesses that miss the CPU caches are served from local DRAM and the rest from CXL, the average memory latency is:

L(h) = h x 100 + (1 - h) x 250

h = 0.95 gives 107.5 ns (7.5% worse than all-local)
h = 0.90 gives 115 ns (15% worse)
h = 0.80 gives 130 ns (30% worse)
h = 0.50 gives 175 ns (75% worse)

The average hides the structure. Workloads that are latency-bound with dependent loads, such as pointer-chasing B-trees or hash probes, see the average almost directly. Workloads that overlap many outstanding misses, such as streaming scans, are bandwidth-bound and barely care. And application-level slowdown is far smaller than memory-latency slowdown, because the CPU spends part of its time not waiting on DRAM.

The Pond paper quantifies the spread for real applications: at 182% latency, 26% of 158 workloads slowed by under 1%, another 17% by under 5%, and 21% by more than 25%. At 222%, the share above 25% rose to over 37%. So the question is never “does CXL slow my system” but “which fraction of my pages and which of my workloads fall into the tolerant group.”

Capacity cost model

Suppose a service needs 2 TB of memory per host. Option A uses 32 x 64 GB DIMMs on a 2-socket, 32-slot platform, which is possible but may force a high-end platform. Option B uses 1 TB of local DRAM plus 1 TB on CXL expanders built from cheaper or reused DDR modules. Define the price per gigabyte of the local DIMMs as P, and of the expander capacity as r x P, where r includes controller and card overhead amortized over capacity.

Cost A = 2048 P. Cost B = 1024 P + 1024 r P = 1024 P (1 + r). Option B wins when 1 + r < 2, that is r < 1. The expander card has a fixed controller cost, so r depends on how much capacity each card carries and which DIMM density you use. The advantage usually comes from using lower-cost DIMM densities on the card, or from avoiding the platform upgrade that Option A would force. If DIMM pricing is roughly linear in capacity, there is little gain; the gain exists where the 128 GB and 256 GB DIMMs carry a premium per gigabyte. Check current DIMM quotes, because this dynamic changed across the 2024 to 2026 DRAM pricing cycle.

At fleet scale, the more interesting argument is Microsoft’s. The Pond paper reports that around 50% of VMs never touch more than half of their rented memory, and that stranding hits about 6% of memory at 75% core utilization and over 10% at around 85%. With pools of 8 to 16 sockets, the arXiv version of the paper reports roughly 9% DRAM reduction at the 182% latency point, and an overall cloud server cost reduction of about 3.5%. The SemiAnalysis write-up of an earlier version of the same work quotes 9 to 10% DRAM reduction and a 4 to 5% server cost reduction. The figures differ between versions, so quote them as “single-digit percent of server cost” rather than a precise value.

Bandwidth sanity check

A host with eight DDR5-4800 channels has a theoretical 307 GB/s. Four x8 CXL expanders add at most about 126 GB/s raw (4 x 31.5), and in practice less. If your workload is bandwidth-bound and you move half its pages to CXL, you may gain capacity and lose throughput. For bandwidth-bound AI preprocessing or analytics, use interleaving only if you have measured a gain.

Workload Fit: Databases, Vector Search and KV Cache Offload

The short version: CXL memory fits workloads with a large, skewed working set where most of the data is cold or read-mostly, and it fits poorly where every access is latency-critical and uniformly random.

Figure 4: Life of a page. Allocated in the fast tier, demoted under pressure, served at higher latency when touched, and promoted back after the hint fault counter marks it hot.

In-memory databases and caches

In-memory stores such as Redis, Memcached and SAP HANA-style column stores often have strongly skewed access patterns: a small hot set and a long tail. Tiering puts the tail on CXL. The database research community has argued that CXL enables “scale-up” engines again, since a single node can hold tens of terabytes; see CXL and the Return of Scale-Up Database Engines. The key gotcha is that indexes and hash tables are pointer-heavy and sensitive to latency, so you want them in DRAM and let bulk row or value storage drift to CXL.

For disk-based engines, the same skew applies to the buffer pool. If you run an LSM-tree store, the block cache and memtables compete for memory; see our post on LSM-tree compaction and write amplification in RocksDB for how memory budget interacts with read amplification. A larger block cache on CXL turns some SSD reads into 200 to 300 ns memory reads, a large win versus microseconds-scale flash, even though it is slower than DRAM.

Vector databases

Vector search workloads such as HNSW graphs have random access over a large dataset, and graph traversal is a chain of dependent loads. That makes them latency sensitive per hop, but recall-latency budgets are often in milliseconds, which tolerates some slowdown. The practical pattern is to keep the upper graph layers and hot centroids in DRAM and place raw vectors, which are read at the end of a search for re-ranking, on CXL. Quantized in-memory indexes can reduce footprint so much that CXL is unnecessary; compare with the techniques in the vector-index comparison on this site before buying hardware.

KV cache offload for LLM serving

Inference servers hold attention key-value caches that grow with context length and concurrency. GPU HBM is the scarce resource, and offloading cold KV blocks to host memory is standard practice in systems such as the one analyzed in our post on NVIDIA Dynamo, LMCache and disaggregated LLM serving. CXL adds a larger, cheaper host-side tier below local DRAM. A KV block reloaded from CXL over PCIe to the GPU takes a path whose bottleneck is typically the PCIe link to the GPU, not the CXL latency, so CXL can work well as a capacity tier for prefix caches that are reused across requests. I do not have verified benchmark numbers for CXL-backed KV offload in production, so treat claims here as architectural reasoning rather than measured results.

Workloads that should stay local

Avoid CXL for latency-critical, uniformly hot data such as lock tables, small hot indexes, real-time control state, and anything with tight tail-latency SLOs and no skew. Also avoid it for bandwidth-bound HPC kernels unless interleaving is tested. And avoid it where your DRAM is not actually the constraint: if cores or network saturate first, extra capacity does nothing.

Decision matrix

Workload	Access skew	Latency sensitivity	CXL fit	Placement advice
In-memory KV store, large values	High	Medium	Good	Index in DRAM, values tiered
Column store analytics	Medium	Low (scan)	Good	Bulk columns on CXL
OLTP hot indexes	Low	High	Poor	Keep local, bind explicitly
Vector search, raw vectors	Medium	Medium	Moderate	Graph local, vectors on CXL
LLM KV prefix cache	High	Low (PCIe bound)	Good	Cold blocks on CXL
Streaming HPC	Low	Bandwidth	Poor	Test interleave only
VM hosting with idle memory	High	Varies	Good	Pool, predict untouched memory

Trade-offs, Gotchas and What Goes Wrong

Honest assessment: CXL memory is real and deployable, but it has sharp edges.

Latency tails. Averages hide the tail. A page that flips between tiers, because promotion and demotion thresholds are too close, burns CPU and bandwidth migrating. This is the classic ping-pong problem, and it is the reason the kernel exposes a promotion rate limit. Set it conservatively and watch pgpromote_success and pgdemote_* counters in /proc/vmstat.

Failure domains grow. Local DIMMs fail with the CPU’s memory controller; a CXL device is a separate component with its own firmware, link and power. A link error, device reset or hot-removal can take memory away from running processes. Because the default online_movable memory can be migrated away, it is easier to remove than kernel memory, but if applications hold pinned or non-movable pages there, removal can fail or crash a workload. Treat the CXL device as a disposable tier: no kernel allocations, no pinned DMA buffers, and no data you cannot afford to lose from a crash.

RAS and error handling. CXL 4.0 explicitly includes enhanced memory RAS, which tells you RAS was a gap. Poison handling, error reporting and device firmware update paths are less mature than for DIMMs. Verify how your platform reports uncorrectable errors and whether the OS can isolate bad pages. This is where I would demand vendor documentation, not marketing.

Pooling adds a control plane. A pool needs a fabric manager, security and isolation between tenants, and policies for rebinding. If a host fails while holding pool capacity, you must release it cleanly. Production pooling remains mostly at research and early-deployment stage; the Pond results are modeled on emulated latency, not on fielded switches. I am not aware of verified public data on large-scale production pooling with CXL switches, so do not assume it.

Sharing is not a free lunch. Multi-host coherent sharing in CXL 3.x means software must handle consistency and failure. Few applications are ready for that.

Anti-patterns. Buying capacity without measuring the working set; leaving numa_balancing off and expecting promotion; binding hot data to CXL; ignoring that a x8 card has less bandwidth than one DDR channel; and assuming a lab latency number carries to a different controller.

Practical Recommendations

Start from measurements, not from the spec. Profile your working set with page-level tracing (for example DAMON, the kernel’s data-access monitor, or perf mem) and compute what fraction of resident memory was touched in the last minutes. If less than half is warm, tiering has headroom; if nearly all is hot, spend money on DRAM.

Pilot with a single expander per host and a workload with a large cold tail. Use the latency model above with your measured CXL latency and expected hit fraction, then compare against the application-level SLO instead of memory latency. Decide whether you want automatic tiering or explicit binding for each component.

Plan for failure. Use movable-only onlining, keep kernel allocations local, and design the service to restart cleanly if the tier disappears. For pooling, wait until your vendor can show a fabric manager, isolation model and RAS story; evaluate it as an infrastructure project, not a drop-in card.

Checklist:

Measure working set and access skew (DAMON or equivalent) before purchase.
Measure CXL latency and bandwidth with your own tool; compare to HMAT values.
Confirm CPU and BIOS CXL 1.1 or 2.0 support and kernel version with memory tiering.
Online memory as movable; keep kernel and pinned memory local.
Enable demotion; set numa_balancing to the memory-tiering mode and tune the promotion rate limit.
Watch promotion and demotion counters for ping-pong.
Bind latency-critical structures explicitly to local DRAM.
Test device removal and error injection before production.
Re-do the cost model with current DIMM and card pricing.

Frequently Asked Questions

What is CXL memory?

CXL memory is DRAM attached to a server through a Compute Express Link device, usually a Type 3 device using CXL.io and CXL.mem. It rides on PCIe lanes but offers cache-coherent, byte-addressable load/store access, so the OS can use it as regular RAM. It is slower than local DIMMs, so it is normally used as a second memory tier for capacity rather than as a replacement.

Is CXL memory slower than DDR5?

Yes. A direct-attached CXL expander adds latency from the link and device controller. Research such as the Pond paper models roughly 1.8 to 2.2 times local DRAM latency for small pools, with larger penalties for switched rack-scale pools. Bandwidth is also bounded by link width: an x8 PCIe 5.0 link is about 31.5 GB/s raw per direction, below one DDR5-4800 channel. Exact numbers depend on the product.

What is the difference between memory pooling and memory tiering?

Tiering is an operating-system policy that places hot pages in fast memory and cold pages in slower memory on the same host. Pooling is a hardware and fabric feature that shares a bank of memory devices among several hosts through a switch, assigning capacity dynamically. You can use both: a host can tier between local DRAM and pooled CXL memory.

Which Linux kernel version do I need?

Memory-tier and demotion support arrived in stages. Most of the TPP patches were merged in Linux 5.18, and the explicit memory-tiers framework came afterwards. CXL region and DAX driver support has kept evolving, so use a recent long-term-support kernel and check your distribution’s documentation for numa_balancing memory-tiering mode and the CXL driver. I have not verified a single minimum version here.

Does CXL 3.0 hardware exist?

CXL 3.0 was released in August 2022 and requires PCIe 6.0 signalling. Mainstream CPUs announced in the first CXL wave supported CXL 1.1, and many current deployments are CXL 1.1 or 2.0 class expanders. Check the specific CPU and device datasheet; I could not verify broad availability of CXL 3.x switches and sharing in production for this article.

Should I use CXL memory for AI workloads?

It can help with capacity-bound parts such as KV cache prefix storage and large embedding or vector tables, where cold data dominates. It does not help GPU HBM bandwidth or compute. Because the GPU-to-host path is usually limited by PCIe, the CXL latency is often secondary there. Validate with your own serving traces, because published production numbers are scarce.

CXL Memory Expansion, Pooling and Tiering: Type 3 Devices, Linux NUMA and Workload Fit (2026)

CXL Memory Expansion, Pooling and Tiering: Type 3 Devices, Linux NUMA and Workload Fit (2026)