Introduction to CXL 3.0
CXL 3.0 is an open-standard interconnect technology that builds upon PCIe 6.0 to facilitate high-speed communication between CPUs and peripheral devices. With a doubled bandwidth of 64 GT/s and enhanced fabric capabilities, CXL 3.0 aims to optimize system-level flows, improve resource utilization, and enable new device types for composable disaggregated infrastructure.
Unveiling the Viral Feature
Viral is an error containment mechanism. CXL links and CXL devices are expected to be Viral-compliant. Viral support capability and control for enabling are reflected in the DVSEC. Viral is not a replacement for existing error-reporting mechanisms. Instead, its purpose is an additional error-containment mechanism.
CXL 3.0 has 68B flit mode and 256B flit mode, the viral procedure of 68B flit mode is the same as CXL 2.0. The major change is CXL 3.0 adding a new viral procedure for standard 256B flit mode and Latency-Optimized 256B flit mode. CXL 3.0 introduces a new type of Link Layer Control Message, Inband Error which has a viral subtype to inject and convey viral status.
Standard 256B Mode Procedure
When the viral condition is detected, the Link layer should signal to the physical layer to stop the currently partially sent CXL.cachemem flit by injection of a CRC/FEC corruption that ensures a retry condition. The Viral control flit is injected as soon as possible after the viral condition is observed. Then the Logical Physical Layer will also remove that flit from the retry buffer and replace it with the Viral control flit that must be sent immediately by the Link Layer. The Link Layer must also resend the flit that was corrupted after the viral flit.
Above is an example of Standard 256B viral procedure. At Cycle “x3”, the Link Layer signaled to corrupt the current flit. At cycle “x4”, the Physical layer corrupts the current flit’s CRC and the Link Layer starts sending the Viral control flit. At Cycle “x5”, Flit-A is removed from the retry buffer and then replaced with the Viral flit sent from the Link Layer. At Cycle “x6”, the Viral flit is also sent with corrupted CRC to ensure the full retry flow. Also starting at cycle “x6”, Flit-A is resent from the Link Layer and forwarded on normally through the Physical Layer and retry buffer. Flit-A is identical to the flit started in Cycle “x2”.
Latency-Optimized (LOpt) 256B Flit Mode Procedure
If the Link Layer is in the first 128B phase of the flit, the flow is identical to the Standard Flit mode. However, if the link layer is in the second phase of the 128B flit, then the flit corruption is guaranteed only in the second half, but the Physical Layer will remove the entire flit from the retry buffer. The Link Layer will send the first 128B identically to what was sent before, and then the Link Layer will inject the Viral control Flit in Slot 8 (HS-format), and Slots 9-14 are considered RSVD.
Above is an example of the second phase of 128B flit’s viral packing flow.
At Cycle “x3”, the Link Layer signaled to corrupt the current flit. At Cycle “x4”, the Link Layer resends FlitA-0 (first half of FlitA) because this half of the flit may have already been consumed. Then at Cycle “x5”, in the second half of that flit, it injects the viral control flit (HS-format). At Cycle “x6”, the FlitA-1 (second half of FlitA) is repacked in the FlitB-0 (first half of FlitB) following the standard packing rules. Starting from Cycle “x4” to Cycle “x6”, the Physical Layer corrupts all the flits’ CRC to ensure full retry flow.
Summary
The key part of Viral in CXL 3.0 is the correct flit packing and CRC corruption, especially for Latency-Optimized mode. Compared with CXL 2.0, CXL3.0 uses a new type of viral control flit and moves retry flow from the Link Layer to the Physical Layer.
For More Information
- For more info on how Cadence PCIe Verification IP and TripleCheck enable users to confidently verify disruptive changes, see Simulation VIP for CXL, Simulation VIP for PCIe, and TripleCheck for PCIe and CXL
- For the press release on CXL3.0, read Cadence's CXL 3.0 Verification IP Press Release
- For more information on CXL in general, visit the CXL Consortium website