The PCIe 6.0 Specification released in 2021 doubles the performance to 64GT/s transfer rate with PAM4(Pulse Amplitude Modulation with 4 levels) modulation and uses FLIT (Flow Control Unit) as the unit of communication for efficiency. In ‘What Disruptive Changes to Expect from PCI Express Gen 6.0? we covered what significant features PCIe 6.0 evolved to embrace.
Amongst many new features and changes in PCIe 6.0, we will talk about one major significant new feature: FLIT. The following mainly touches upon challenges and corresponding solutions based on our design and verification experiences.
What is FLIT in PCIe 6.0?
The transactions in previous versions had a variable length of size, known as TLPs. They may have a fixed header size but had a different length of data payload. No matter how long the TLP is, it is protected by 32-bit CRC. In PCIe 6.0, the additional signal states of PAM4 result in a more fragile signal than an NRZ. The new modulation requires FEC to compensate for PAM4’s higher bit-error rate, and the error correction needs to operate on fixed-sized packets, hence the adoption of FLIT (Flow Control Unit) for PCIe 6.0.
FLIT has a fixed 256-byte length of size which consists of 236-byte of TLP, 6-byte of DLP, 8-byte of CRC and 6-byte of FEC. It removes Sync Header in 1b/1b encoding, framing token, etc. FLIT also has a similar sequence number concept, in which the first 2 bytes of the DLP carry information dedicated to FLIT level sequence number, Ack/Nak, Retry mechanism, etc.
FEC(Forwarded Error Correction) is designed for latency and complexity increases exponentially with the number of symbols corrected. 6 bytes of FEC are responsible for 3 interleave groups and each group has 2 FEC bytes. This is to prevent burst errors if it is smaller than 3 bytes.
The first challenge is regarding new FLIT format and encoding changes.
The FLIT-enabled mechanism and negotiation are happening by the beginning of the link training, Polling and Configuration, using the FLIT Mode Supported bit in the ‘Data Rate Identifier’ field (Symbol 4, Bit 0) in the TS1. Once it is negotiated, it applies to all data rates, implying FLIT is also supported 8b/10b and 128b/130b (Hybrid mode).
In FLIT mode, we are using a completely new TLP Header format. Previous TLP Headers had many limitations, like no room for increasing tag size. PCI-SIG redesigned header to suites suited FLIT mode. The challenge will be to test all the new combinations.
TLP Header is composed of a 3 to 7 DW TLP header Base, followed by 0 to 7 additional DWs of OHC (Orthogonal Header Content).
The new type of field has a fully decoded 8b Packet Type field. This means that all the 256 Type values are defined or reserved for a certain group to permit proper framing and forwarding.
Also, there are new completion rules designed for FLIT mode. The completion for non-posed TLPs also has major updates including a 14-bit tag, error report using OHC-A5, etc.
As we know, 236 bytes can accommodate 1 TLP or many TLPs, or if you have a long TLP, it can be split into multiple FLITs. Also, between each TLP, it may have a NOP TLP if no further TLPs are scheduled. There are rules like no more than 4 TLPs (non-NOP) per FC/VC in the 32 DW boundary of the FLIT, which are also new to the PCIe 6.0 spec.
The DLP is a 6 bytes sequence, and the first 2-bytes are dedicated to FLIT level Ack/Nak, Retry, etc. so there is a new format for it. For instance, the “FLIT usage” decides between IDLE FLIT, NOP FLIT, or payload FLIT; “Prior FLIT” had non-NOP or NOP which is used to avoid retry error; and “replay command” along with “sequence number” decide Ack/Nak/Retry.
Moreover, multiple FLIT transmission and exchange rules are newly defined in the FLIT sequence number and retry mechanism. For example, the CONSECTIVE_TX_EXPLICIT_SEQ_NUM_FLITS and CONSECUTIVE_TX_NAK_FLITS counters and it is relevant rules.
To test all major encoding and format changes, we must ensure all new packet fields are covered. The generic solution for the above new features will be to define and exercise good coverage to test the new feature well. A good coverage model assists to test new features well.
Other than the FLIT format, another big challenge for FLIT is the new sequence number and the Retry mechanism.
One of the most difficult verification challenges is the IMPLICIT_RX_FLIT_SEQ_NUM rules. This counter is essential for the replay mechanism. As the implicit FLIT sequence number is not carried in the FLIT, it all depends on internal logic to handle it. The internal logic/counter needs to handle multiple scenarios to make sure the IMPLICIT_EX_FLIT_SEQ_NUM calculation is correct.
It is essential to make sure that the TX retry buffer is correct as it needs to be stored in all FLITs before receiving Ack or Nak. As multiple TLPs can be in one FLIT or one large TLP can be split into various FLITs, we need to guarantee that the retried FLIT should not skip or add an extra TLP to the original FLIT. It is essential for Posted TLPs as there is no completion for it. Lost TLPs will cause uncorrectable errors.
The new Standard Nak/selective Nak can let the transmitter replay a certain FLIT or multiple FLITs. The relevant rules are impacting both TX and RX retry buffer buffers. Also, sending Standard or selective NAK is implementation-specific, so sometimes it hard to predict and make checking if there are protocol violations.
The FEC algorithm is a new feature that we need to ensure there are no bugs in the calculation on both TX and RX sides.
Based on the above pointers that we have discussed, below are the recommended solutions we have tried to verify our design:
- In addition to following Spec to implement the IMPLICIT_RX_FLIT_SEQ_NUM counter correctly, we found it is necessary and valuable to make implicit FLIT sequence number available in the status register or make it visible in the debug log for easier comparing and debugging.
- Make sure your monitor can store all FLITs and can compare saved FLIT with retired FLIT on every symbol and report the error accordingly.
- You must make both RTL and monitor have a big enough retry buffer in case standard NAK is received, and multiple FLITs will be resent. Also, implement a checker to predict standard and selective NAK transmit conditions based on Spec. Third, to examine if the replayed FLIT’s sequence number matches with the selective NAK received.
- To make sure FEC algorithm logic is implementation is correct, use an Error Injection test case with randomized symbol location needed.
In summary, PCIe 6.0 is a complex protocol with many verification challenges. You must understand many new Spec changes and think about the robust verification plan for the new features and backward compatible tests impacted by new features. Cadence’s PCIe 6.0 Verification IP is fully compliant with the latest PCIe Express 6.0 specifications and provides an effective and efficient way to verify the components interfacing with the PCIe 6.0 interface. Cadence VIP for PCIe 6.0 provides exhaustive verification of PCIe-based IP and SoCs, and we are working with Early Adopter customers to speed up every verification stage.
More Information:
- For more info on how Cadence PCIe Verification IP and TripleCheck VIP enable users to confidently verify PCIe 6.0, see our VIP for PCI Express, VIP for Compute Express Link and TripleCheck for PCI Express
- See the PCI-SIG website for more details on PCIe in general and the different PCI standards.