PCIe designed system fabrics rely on software enumeration by Operating System (OS) for device discovery. CXL 2.0 device is exposed as PCIe native endpoint and CXL 1.1 is exposed as root complex integrated endpoints (RCiEP) during software enumeration process. Notably, there is a difference between the discovery of CXL 1.1 versus the CXL 2.0 device. Hence, the configuration space for CXL1.1 and CXL 2.0 varies. In this blog, we will discuss newly introduced registers for the CXL-compliant devices and how they are discovered during the CXL enumeration flow.
New Register Space with CXL
There are two sets of register space in CXL, one that resides under configuration space and the another set that resides under memory mapped space.
- Configuration space defined register space
- Memory mapped register space
Configuration Space Defined Register Space
All configuration space registers are identified as PCIe defined designated vendor-specific extended capabilities (DVSEC) registers with DVSEC Vendor ID as h1E98 (reserved for CXL). DVSEC ID associated with capability helps determine the type of CXL capability. The key capabilities are:
- PCIe DVSEC for CXL Devices: Important DVSEC carries information for device attributes like semantics supports, host managed device settings, etc.
- GPF DVSEC for CXL Devices: Contains Global Persistent Flush (GPF) feature specific parameters
- PCIe DVSEC for Flex Bus Port: Critical DVSEC consists of particulars on CXL link training
- Register Locator DVSEC: Application-defined register blocks
- MLD DVSEC: Contains Multi-Logical devices (MLD) feature-specific parameters
An important point to note here all register spaces defined above exist in PCIe-defined configuration space for CXL2.0 devices. Whereas for CXL1.1, DVSEC7 (PCIe DVSEC for FlexBus Port) resides in Root Complex Register Blocks (RCRB) space and rest others reside in RciEP defined configuration space. Hence, to access these registers software should use CXL.io memory read/write cycles for RCRB space-defined register else CXL.io configuration read/write cycles.
Memory Mapped Register Space
CXL Memory mapped registers are in one of the PCIe-defined Base Address Ranges (BARs) for CXL2.0 mode whereas for CXL1.1 the same registers are defined in MEMBAR0 regions of RCRB space. These memory locations specify component registers:
- io registers: Reserved for CXL.io operations
- cachemem registers: Contains CXL-defined capabilities like Link, IDE, HDM, RAS, etc. Link capability is a critical one that governs link layer operations like credit flow, replay etc.
- CXL ArbMux registers: Consists of registers predicting weightage for CXL.io versus CXL.cachemem traffic
Devices Operating in CXL 1.1 Mode with No RCRB
Devices operating in CXL1.1 mode with no RCRB ECN were introduced with CXL 2.0 and included in CXL 3.0 specification as well. As discussed earlier, some registers in CXL 1.1 are visible under configuration space, RCRB, and MEMBAR0 space, while the same registers are located under endpoint configuration space and PCIe BARs for CXL2.0 compliant devices. In this case, CXL 2.0 device needs to morph register space dynamically depending on alternate protocol negotiation results. This ECN simplifies operations between the CXL 1.1 host and CXL 2.0 devices and eliminates the need for dynamic register mapping. The mechanism defined does not have any hardware implication and adds the requirement of adding memory read requests during enumeration to identify where component registers exist.
More information:
- For more info on how Cadence PCIe Verification IP and TripleCheck enables users to confidently verify disruptive changes, see Simulation VIP for CXL, Simulation VIP for PCIe and TripleCheck for PCIe and CXL
- For the press release on CXL3.0, read Cadence CXL 3.0 Verification IP Press Release
- For more information on CXL in general, visit the CXL Consortium website