CXL (Compute Express Link): Memory Expansion and Coherency
Compute Express Link (CXL) is an open, industry-standard interconnect that delivers low-latency, cache-coherent communication between a host processor and attached devices such as accelerators, memory expanders, and smart I/O. Built on the PCI Express physical layer, CXL solves a problem that PCIe alone cannot: it lets the CPU and a device operate on the same data with hardware-managed coherency, while also enabling memory to be expanded, pooled, and shared across a rack. This guide covers the protocol stack, device types, the coherency model, and the datacenter use cases driving adoption.
Quick Summary
| Foundation | Runs on the PCIe PHY (CXL 1.x/2.0 = PCIe 5.0, CXL 3.x = PCIe 6.0 PAM4) |
| Three Protocols | CXL.io (discovery/config), CXL.cache (device caches host memory), CXL.mem (host accesses device memory) |
| Key Value | Hardware cache coherency plus byte-addressable memory expansion, pooling, and sharing |
| Evolution | 1.1 (direct attach) → 2.0 (switching, pooling) → 3.x (fabrics, multi-level switching, sharing) |
Why CXL Exists
Modern datacenter servers face two structural problems. First, memory is stranded: DRAM is bolted to individual CPU sockets, so a server with spare memory cannot lend it to a neighbor that is memory-starved. Second, accelerators and CPUs cannot efficiently share data: moving data across plain PCIe requires explicit DMA copies and software-managed consistency, which adds latency and CPU overhead.
CXL addresses both. By layering coherent protocols on top of the ubiquitous PCIe electrical and form-factor ecosystem, it allows a device to participate in the host's coherency domain and allows host memory capacity to scale beyond the limits of the CPU's native DDR channels. Crucially, CXL preserves a load/store, byte-addressable memory model, so software treats expansion memory like ordinary RAM rather than block storage.
Built on the PCIe Physical Layer
CXL does not reinvent the wire. It reuses the PCIe SerDes, link training, and connectors, so CXL devices plug into the same slots and use the same signaling as PCIe.
- Shared PHY: CXL 1.1 and 2.0 operate over PCIe 5.0 at 32 GT/s per lane; CXL 3.0/3.1 move to PCIe 6.0 at 64 GT/s using PAM4 signaling.
- Alternate Protocol Negotiation: During PCIe link training, the two ends negotiate via a flex-bus mechanism. If both support CXL, the link runs CXL protocols; otherwise it falls back to standard PCIe.
- Common Form Factors: CXL devices use familiar add-in cards, U.2, and the EDSFF E3.S/E1.S modules common in servers.
- Latency Focus: Unlike PCIe transaction-layer packets, CXL.cache and CXL.mem use a compact 68-byte (or 256-byte in CXL 3.0) flit format to minimize round-trip latency.
The Three CXL Protocols
A single CXL link multiplexes three distinct sub-protocols over the same physical lanes. Each serves a different access pattern, and a given device implements some subset depending on its role.
CXL.io
CXL.io is functionally equivalent to PCIe and is mandatory on every CXL link. It handles device discovery, enumeration, configuration register access, interrupts, DMA, and error reporting. Without CXL.io a device cannot be initialized, so it forms the control plane for the coherent protocols.
CXL.cache
CXL.cache lets a device coherently cache host memory. An accelerator can read and write data that lives in the host's DRAM while hardware keeps its local cache consistent with the CPU's view. This eliminates explicit DMA copies for fine-grained, latency-sensitive accelerator workloads. It is an asymmetric protocol: the host owns the coherency directory, and the device issues requests (such as RdShared or RdOwn) and responds to snoops from the host.
CXL.mem
CXL.mem lets the host directly access memory attached to the device using simple load/store semantics. The host issues read and write commands to the device's memory controller, which may front DDR DRAM, persistent memory, or other media. This is the protocol behind memory expansion and pooling, and it presents device memory to the OS as additional system RAM.
CXL Device Types
The specification defines three device types, distinguished by which protocols they implement and whether they expose their own memory to the host.
| Device Type | Protocols Used | Has Own Memory? | Typical Examples |
|---|---|---|---|
| Type 1 | CXL.io + CXL.cache | No (caches host memory) | SmartNICs, NICs with caches, accelerators without local memory |
| Type 2 | CXL.io + CXL.cache + CXL.mem | Yes (host-managed coherent) | GPUs, FPGAs, AI/ML accelerators with HBM/local DRAM |
| Type 3 | CXL.io + CXL.mem | Yes (exposed as system memory) | Memory expanders, memory pooling appliances, persistent memory modules |
Type 1 devices are coherent accelerators that operate on host data. Type 2 devices add their own memory that the host can also access coherently, ideal for accelerators that need both to cache host data and to expose local HBM. Type 3 devices are the workhorses of memory expansion and pooling, contributing capacity and bandwidth without caching host memory themselves.
The Coherency Model: Bias-Based Coherence
For Type 2 devices, which contain memory that both the device and the host want to use, CXL employs a bias-based coherency model. Rather than constantly arbitrating every access through a single shared directory, the model assigns each page of device-attached memory a coherence "bias" that optimizes for the expected access pattern.
Host Bias
In host bias mode, coherency for the device's memory is managed by the host's coherency engine. The host can freely cache that memory, and when the device accesses it, requests are routed through the host to ensure consistency. This mode is used when the host (CPU) is actively producing or consuming the data, for example while staging inputs before an accelerator run.
Device Bias
In device bias mode, the device is guaranteed it can access its own attached memory without sending coherency requests to the host, because the host is known not to hold cached copies. This delivers near-local latency and is used during the compute phase when the accelerator is churning through its working set. Software or the device driver flips a page between host bias and device bias as the workload moves between staging and compute phases, avoiding the overhead of snooping the host on the device's hottest path.
Memory Expansion, Pooling, and Sharing
Memory Expansion
The most immediate use of CXL is adding memory capacity and bandwidth beyond the CPU's native DDR channels. A Type 3 expander on a CXL slot presents its DRAM to the OS as an additional, typically CPU-less NUMA node. Operating systems and tiering software place hot data in local DDR and cooler data in CXL memory, scaling per-socket capacity into the multi-terabyte range without adding sockets.
Memory Pooling
Introduced with CXL 2.0, pooling places a shared block of memory behind a CXL switch so that it can be dynamically allocated to multiple hosts. A pooled device supports Multiple Logical Devices (MLD), partitioning its capacity into logical slices that an orchestrator assigns to whichever server needs them. This directly attacks memory stranding: instead of over-provisioning every server for its worst case, the datacenter sizes a shared pool and hands out capacity on demand.
Memory Sharing
CXL 3.0 extends pooling to true sharing, where the same region of memory can be coherently accessed by multiple hosts at once. Backed by hardware-enforced coherency across the fabric, this enables shared in-memory data structures and low-latency message passing between servers without copying data over the network, a foundation for genuine memory disaggregation.
CXL Generations: 1.1, 2.0, and 3.x
Each CXL generation broadens the topology, from a single device on a link to a switched fabric spanning a rack.
| Version | PCIe Base | Headline Capabilities |
|---|---|---|
| CXL 1.1 | PCIe 5.0 (32 GT/s) | Direct-attach single device per link; the three protocols; coherent accelerators and basic expansion |
| CXL 2.0 | PCIe 5.0 (32 GT/s) | Single-level switching, memory pooling with MLD, hot-plug, and link-level integrity and data encryption (IDE) |
| CXL 3.0 / 3.1 | PCIe 6.0 (64 GT/s, PAM4) | Multi-level switching, fabrics with up to 4096 nodes, memory sharing, peer-to-peer, and 256-byte flits |
From Switching to Fabric
CXL 2.0 introduced a single layer of switching, enabling one switch to fan out to multiple devices and to carve pooled memory among hosts. CXL 3.0 doubles the raw bandwidth with PCIe 6.0 PAM4 signaling and introduces multi-level switching and a true fabric: a port-based routing scheme that lets thousands of hosts and devices interconnect, communicate peer-to-peer, and share memory. This is the architectural basis for composable, disaggregated infrastructure where compute and memory are independently scaled resources.
Latency Expectations
CXL Memory Latency vs Local DRAM
A direct local DDR5 read typically completes in roughly 80–140 ns. CXL-attached memory adds the overhead of the link controllers, SerDes, and flit framing on top of the underlying media.
Direct-attached CXL Type 3 expander: approximately 170–250 ns, comparable to accessing a remote NUMA node across a socket interconnect.
Switched / pooled CXL: add roughly 50–100 ns per switch hop.
Because CXL memory is slower than local DRAM, it is best used as a capacity and bandwidth tier for warm data, with hot pages kept in local DDR by the OS or memory-tiering software. The byte-addressable model still makes it far faster than any storage-class alternative.
Datacenter Use Cases
Memory Disaggregation
- Pooled Capacity: Right-size memory at the rack level instead of per server, reclaiming stranded DRAM and cutting total cost of ownership.
- Composable Infrastructure: Allocate memory to hosts on demand so that a database node and a cache node draw from the same shared pool.
- In-Memory Databases: Scale working sets into the multi-terabyte range without expensive high-DIMM-count servers.
- AI/ML Acceleration: Type 2 accelerators coherently share data with the host and expand model and KV-cache capacity through CXL memory.
- Shared Data Structures: CXL 3.0 memory sharing enables coherent, low-latency data exchange between servers without network copies.
Implementation Best Practices
- Choose the Right Device Type: Map the workload to Type 1 (coherent compute on host data), Type 2 (accelerator with local memory), or Type 3 (pure expansion/pooling) before committing to silicon or IP.
- Plan NUMA and Tiering: Expose CXL memory as a distinct NUMA node and pair it with OS or software tiering so that hot pages stay in local DDR and cold pages migrate to CXL.
- Manage Bias Transitions: For Type 2 designs, flip pages between host bias and device bias around staging and compute phases to keep the accelerator's hot path off the host snoop path.
- Budget for Link Latency: Account for added round-trip latency and per-hop switch cost in performance models; avoid placing latency-critical data structures in pooled memory.
- Enable Link IDE: Use CXL Integrity and Data Encryption for confidentiality and tamper detection, especially across pooled and shared topologies.
- Validate Interop and Fallback: Confirm flex-bus negotiation, PCIe fallback, and hot-plug behavior across host BIOS, switch, and device firmware early in bring-up.
- Verify Coherency Rigorously: Exercise snoop, ownership, and bias-flip corner cases with directed and random coherency tests before production.
Conclusion
CXL turns the PCIe slot into a coherent memory and accelerator fabric. Its three protocols—CXL.io, CXL.cache, and CXL.mem—combine with three device types and a bias-based coherency model to enable coherent acceleration, byte-addressable memory expansion, and rack-scale pooling and sharing.
As CXL advances from direct attach in 1.1 to switching in 2.0 and full fabrics in 3.x, it is becoming the backbone of composable, disaggregated datacenters where memory and compute scale independently. Designers who understand device types, the coherency model, and realistic latency budgets are positioned to extract the most value from the technology.
Vcores offers silicon-proven CXL and PCIe IP cores along with controller integration, coherency verification, and design services to accelerate your datacenter and accelerator SoC projects.
