Overview
The PCIe protocol evolved to its sixth generation in 2021, doubling its transfer rate to 64 GT/s compared to the previous generation and bringing new features and optimizations to move dependency from software to hardware.
The PCIe hierarchy can be treated as a tree that is rooted in the root port. Each hierarchy can have up to 256 buses. Each bus can host 32 devices, and each device can host eight functions. The tuple of bus/device/function numbers (or just bus/function numbers, in alternative routing ID case) are unique within a hierarchy. In some contexts, the hierarchy is also called a segment.
Does Segmentation Exist Before PCIe 6.0?
Before PCIe 6.0, the segment number was not part of transactions, and it was purely a software concept. To make this routing automatic, with less software intervention, in and after PCIe 6.0, the segment number concept is added into a transaction. So, all config packets, specific message packets, and completion packets contains target segment number.
As per the PCIe spec for non-flit mode, all EP functions should capture the bus and device number from the received Type0 config write request. Similarly, in flit mode, an EP function will also capture its own segment number from the received Type0 config write request internally. This results in each function being uniquely identified by Seg:Bus:Dev:Func. Let’s discuss segmentation more with the figure below.
The above figure shows segmentation in PCIe where:
- Two CPUs connected with an interconnect
- Two different Root Complex (RC) hierarchy structures labeled as segment 0 and segment 1
- Switch in both segments contains one Upstream Port (USP) and two Downstream Port (DSP)
- Each segment has a bus number from 0 to 4
Advantages of Capturing Segment Number
1. For ID-based routing of message TLPs, if a function wants to target other functions in a different hierarchy, it is only possible to do so using destination segment information.
Example: In above example figure, segment 1 EP1(4:0:0) wants to drive the message packet targeting segment 0 EP0 (3:0:0). Then, the generated packet will have the destination segment value set to 0 and the Destination Segment Valid (DSV) value set to 1. So, the Segment 1 switch upstream port will identify that this packet belongs to a different segment and it should be routed to upstream instead of routing internally to the downstream port. The figure below shows OHC-A4 content with byte 0 indicating the destination segment information and byte 2 indicating the DSV field.
Verification challenges: The DSV field poses a major verification challenge as corrupting this bit may change the TLP path, causing it to reach a non-intended endpoint. The verification engineer needs to develop test cases around these to ensure the correct routing of packets. Infact, the verification strategy should include an end-to-end scoreboard as well.
2. Memory requests are address-based routing. Consider the scenario where a non-posted memory read request (MRd) targeted to a specific address falls into a different hierarchy. The completion associated with this non-posted transaction should be routed to the proper segment. For this reason, memory request packets should contain the requester segment number, which can be done by including the OHC-C DW. An OHC-C with a substream field set to ‘b111 means that the associated TLP is a non-encrypted packet. 
Example: In the above example, segment 0 EP0 (3:0:0) generates an MRd packet targeting to the segment 1 address range. Requester segment information in MRd packet helps the associated completion to route back to segment 0. The Requester Segment Valid (RSV) field must be set to 1 here.
Verification challenges: The RSV field poses a major verification challenge as corrupting this bit may change the associated ccompletion path, and the requester will never receive the completion.
3. Providing Segment information can also helpful for logging error information per segment in Advanced Error Reporting (AER) capability structure. The verification testplan should include testcases for AER as well.
Conclusion
The arrival of segment number in PCIe 6.0 flit mode represents a significant leap forward for complex systems. By embedding segment identification within the TLP header, switches can now make informed routing decisions, eliminating the need for the root complex to take ownership of peer-to-peer traffic across segments. This translates to reduced complexity, lower implementation costs, and ultimately, improved performance for data flowing between devices within a system. While legacy non-flit mode communication may still require root complex involvement, software configuration can optimize flit-mode communication for maximum efficiency. This shift towards segment-aware routing paves the way for a new chapter in efficient data flow within the ever-evolving world of PCIe technology.
More Information
- For more info on how Cadence PCIe Verification IP enables users to confidently verify switches, see our VIP for PCI Express
- For more information on PCIe in general and on the various PCI standards, see the PCI-SIG website.