
GPUDirect RDMA, RoCE Offload and ASAP²: What ConnectX Features Do
20 November, 2026
Technical Guide
Short answer: GPUDirect RDMA lets a ConnectX adapter read and write GPU memory directly over PCIe, so network transfers no longer stage through host memory. RoCEv2 offload means the adapter runs the RDMA transport itself over routable UDP/IP Ethernet. ASAP² moves the Open vSwitch data plane into the adapter’s embedded switch. PFC, ECN and DCQCN keep a RoCE fabric lossless and stable under load, and VPI lets the same card run either InfiniBand or Ethernet. Which of these matter depends on whether the node trains, serves or stores.
The ConnectX-7 and ConnectX-8 product pages list these features; this article explains what each one does, based on NVIDIA’s datasheets, manuals and driver documentation.
What RDMA changes
NVIDIA’s driver documentation defines RDMA as direct memory-to-memory transfer between servers without involving the CPU, and notes that ConnectX adapters offload the RDMA services to hardware to reduce CPU load and latency. In practice the adapter, not the kernel network stack, runs the transport and places data in the target memory.
On InfiniBand this is native. The ConnectX-7 InfiniBand datasheet lists a hardware-based reliable transport, RDMA and send/receive semantics, atomic operations, and the XRC and DCT transports. RoCE carries the same InfiniBand transport inside Ethernet packets, and applications on both use the same verbs API.
One operational detail: because the hardware handles RoCE traffic, it does not appear in the standard Ethernet interface counters; NVIDIA points to /sys/class/infiniband/<device>/ports/<port>/counters/ instead.
GPUDirect RDMA: the data path
GPUDirect RDMA, introduced with Kepler-class GPUs and CUDA 5.0, “enables a direct path for data exchange between the GPU and a third-party peer device using standard features of PCI Express”, according to NVIDIA’s CUDA documentation. Without it, data going from GPU memory to the network is first copied into host memory and the adapter reads it from there. With it, the adapter reads from and writes to GPU memory directly.
On the software side, the NVIDIA GPU driver ships the nvidia-peermem kernel module. It gives ConnectX adapters direct peer-to-peer read and write access to GPU memory, is supported on ConnectX-3 VPI and newer adapters, and works with both InfiniBand and RoCE. Both the ConnectX-7 InfiniBand datasheet and the ConnectX-8 datasheet list GPUDirect RDMA.
The hardware side is where projects go wrong. NVIDIA’s documentation is explicit about PCIe topology:
- The GPU and the adapter must share the same upstream PCIe root complex.
- A path through PCIe switches only is optimal and yields the best performance.
- A path through a single CPU works, but performs worse.
- A path that crosses the inter-socket link may be extremely performance-limited or may not work reliably.
The same documentation notes that the IOMMU must be disabled or set to pass-through translation for GPUDirect RDMA to work. NCCL exposes the topology rule directly through NCCL_NET_GDR_LEVEL: PIX uses GPUDirect RDMA only when GPU and NIC share a PCIe switch, PXB across multiple PCIe switches, PHB within the same NUMA node, and SYS even across the inter-socket link. If it is not set, NCCL picks a value for the platform. Card placement is therefore a server design decision, made before the order, not after.
GPUDirect Storage and NVMe-oF
GPUDirect Storage (GDS) applies the same idea to storage: a direct DMA path between GPU memory and storage that avoids a bounce buffer through the CPU, accessed through the cuFile API. For local storage, the NVMe device performs the DMA. For remote storage, the NIC performs RDMA, and GDS for distributed file and block systems relies on GPUDirect RDMA for the direct path between the NIC and GPU memory.
So on a GPU node reading from a parallel file system, GDS depends on the adapter’s GPUDirect RDMA support; the ConnectX-7 InfiniBand and ConnectX-8 datasheets both list GPUDirect Storage. NVIDIA’s GDS support matrix names partner file systems such as DDN EXAScaler, IBM Storage Scale, WekaFS and VAST, and its classification table includes BeeGFS and NFS. The GDS release notes mark NVMe-oF as not functional with some MLNX_OFED versions, so check the driver version before deployment.
NVMe over Fabrics (NVMe-oF) carries NVMe commands over a network such as Ethernet or InfiniBand. Its target offload is a storage server feature. With NVMe-oF target offload, supported from ConnectX-5 onward, the adapter processes regular I/O requests itself and sends them to the NVMe drive over peer-to-peer PCIe. NVIDIA states that, apart from connection management and error flows, no CPU utilization is observed during NVMe-oF traffic. The ConnectX-7 datasheets list NVMe-oF target offloads, NVMe-oF over TCP and RoCE acceleration, T10-DIF signature handover, and storage protocols including SRP, iSER, NFS over RDMA and SMB Direct.
RoCEv2 hardware offload: what the NIC does
RoCEv2 puts RDMA inside UDP/IP packets with destination port 4791. The IP header makes it routable across Layer 3 networks, and the UDP source port can serve as a flow identifier for ECMP load balancing. RoCEv1, by contrast, uses a dedicated EtherType and stays inside one broadcast domain.
“RoCE offload” means the ConnectX adapter implements the RDMA transport for these packets in hardware, as it does on InfiniBand. In the ConnectX-7 manual’s words, by using data center bridging and the adapter’s hardware congestion control, RoCE provides low-latency RDMA services over Layer 2 and Layer 3 networks. The ConnectX-8 datasheet lists RDMA and RoCEv2 accelerations.
RoCE has no subnet manager and no LID, so path records cannot be queried from an SM; NVIDIA recommends the RDMA connection manager (RDMA CM) to handle this. Lossless behaviour and congestion handling have to be configured on the Ethernet side. The trade-offs are covered in RoCEv2 and Ultra Ethernet: a real alternative to InfiniBand? and InfiniBand vs Ethernet.
ASAP²: what it accelerates and when it matters
ASAP² (Accelerated Switching and Packet Processing) is an Ethernet feature for software-defined networking. The problem, in NVIDIA’s words: Open vSwitch runs in the hypervisor and switches in software, which is CPU-intensive and prevents full use of the available bandwidth. ASAP² handles the OVS data plane in the adapter’s embedded switch (eSwitch), supported from ConnectX-5 onward, while the OVS control plane stays unchanged.
The classic setup is built on SR-IOV: virtual functions are passed through to VMs, the adapter’s eSwitch is put into switchdev mode, which creates a representor device per VF, and OVS hardware offload, off by default, is enabled. The ConnectX-8 datasheet lists what it covers: OVS acceleration, overlay networks (VXLAN, GENEVE, NVGRE), connection tracking (L4 firewall) and NAT, hierarchical QoS, header rewrite, flow mirroring, flow statistics and flow aging.
It matters on hosts that run virtual switching: VMs on SR-IOV, multi-tenant private clouds, and network functions with VXLAN overlays. On a bare-metal GPU training node that only talks to an InfiniBand fabric, ASAP² does nothing useful.
QoS and congestion control (PFC, ECN, DCQCN)
InfiniBand. QoS is part of the architecture. Packets carry a service level (SL), switches map SL to virtual lanes (VL), and the subnet manager applies the policy across the fabric. The ConnectX-7 InfiniBand datasheet lists 8 virtual lanes plus VL15, hardware-based congestion control and adaptive routing support.
Ethernet with RoCE. Three mechanisms work together:
- PFC (IEEE 802.1Qbb) pauses a single traffic class instead of the whole link. NVIDIA’s example is lossless service for RoCE and best-effort service for ordinary traffic on the same port. PFC has to be enabled on the switch as well as the host.
- ETS and trust mode decide how traffic classes share bandwidth and how packets are classified. ConnectX adapters trust the VLAN PCP field by default and can be set to trust DSCP instead.
- ECN signals congestion with two bits in the IP header without dropping packets, and every node on the path must support it.
DCQCN is the rate-control loop built on ECN. In NVIDIA’s description the switch is the congestion point and marks ECN with a probability that rises as its egress queue grows. The receiving NIC, the notification point, generates a congestion notification packet (CNP) in hardware for the affected queue pair. The sending NIC, the reaction point, cuts its rate and then recovers it in stages. Newer NVIDIA documentation describes DCQCN as the legacy-interoperability algorithm on ConnectX SuperNICs and BlueField DPUs, alongside ZTR-RTT, which adds hardware-timestamped round-trip time measurement. The ConnectX-8 datasheet lists programmable congestion control.
The practical point behind “end-to-end QoS” is that none of this works if the host and switch disagree. Priority mapping and PFC classes have to be consistent between adapters and switches, and every node on the path must support ECN. That is configured on the RoCE switches as much as on the adapters.
VPI: one card, InfiniBand or Ethernet
ConnectX-7 cards come in two kinds: those that support both InfiniBand and Ethernet, and Ethernet-only models. The manual lists some dual-port QSFP112 and quad-port SFP56 models as Ethernet-only, so the part number matters. On the dual-protocol cards, which NVIDIA calls VPI, the port type is set with mlxconfig (LINK_TYPE_P1, LINK_TYPE_P2) or from UEFI, and takes effect after a reboot.
Defaults differ by part number. According to the ConnectX-7 manual, MCX75310AAS-NEAT ships as InfiniBand NDR, while MCX755106AS-HEAT and MCX755106AC-HEAT ship as 200GbE Ethernet and MCX715105AS-WEAT as 400GbE Ethernet. On ConnectX-8, the single-port OSFP card defaults to InfiniBand and the dual-port QSFP112 card to Ethernet, with Ethernet listed at up to 400 Gb/s per port. Check the default and supported port combinations for your exact part number before cabling.
Which features matter for training, inference and storage nodes
| Node role | Features that matter | Usually irrelevant |
|---|---|---|
| Multi-node training | GPUDirect RDMA, NIC-GPU PCIe placement, lossless RoCE or InfiniBand congestion control, in-network collectives | ASAP², NVMe-oF target offload |
| Inference serving | GPUDirect RDMA if the model spans nodes; SR-IOV and ASAP² if serving runs in VMs or a virtualised Kubernetes network | NVMe-oF target offload |
| Storage servers | NVMe-oF target offload, RDMA storage protocols (iSER, NFS over RDMA, SMB Direct), T10-DIF | GPUDirect RDMA on the server itself |
| GPU clients of parallel storage | GPUDirect Storage, which depends on GPUDirect RDMA | ASAP² |
| Virtualised / multi-tenant hosts | SR-IOV, ASAP², inline crypto | In-network collectives |
For training clusters, the in-network computing side is covered in NVIDIA SHARP: aggregating collectives inside the network. The ConnectX-8 manual notes that ConnectX-8 InfiniBand SuperNICs extend SHARP capabilities.
Mevasis supplies NVIDIA ConnectX adapters and designs InfiniBand and RoCE fabrics, including PCIe placement checks for GPUDirect. Models and quotes are on the InfiniBand adapter category page.
Frequently Asked Questions
GPUDirect RDMA gives a third-party PCIe device such as a ConnectX adapter a direct data path to GPU memory. GPUDirect Storage uses the same idea for file and block I/O through the cuFile API: local NVMe drives DMA into GPU memory, and remote storage reaches it through a NIC using GPUDirect RDMA.
Yes. NVIDIA documents that the nvidia-peermem kernel module, which gives ConnectX adapters peer-to-peer access to GPU memory, works with both InfiniBand and RoCE, on ConnectX-3 VPI and newer adapters.
Usually not. ASAP² offloads the Open vSwitch data plane to the adapter's embedded switch, so it matters on Ethernet hosts that run virtual switching for VMs or containers, not on bare-metal nodes that only talk to an InfiniBand fabric.
The classic RoCE design is lossless Ethernet, and PFC (IEEE 802.1Qbb) is the mechanism that makes a traffic class lossless, enabled on both hosts and switches. NVIDIA also documents Zero Touch RoCE, whose ZTR-RTT congestion control is designed to work without specialized switch configuration, so check which mode your adapters and switches support.
On VPI models, yes. The port type is set per port with mlxconfig or from UEFI and applies after a reboot, and each part number ships with a documented default, for example InfiniBand on the single-port OSFP card and Ethernet on the dual-port QSFP112 cards.
Related Articles
- RoCEv2 and Ultra Ethernet: A Real Alternative to InfiniBand?How RoCEv2 works, why PFC and ECN are critical, and what Ultra Ethernet changes. The technical realities of lossless Ethernet in HPC and AI …
- From Mellanox to NVIDIA Networking: Product Names and Reading OPNsHow Mellanox became NVIDIA Networking, the Quantum and ConnectX generation map, and how to decode MQM, MCS and MSB part numbers when reading …
- HPC Network Design: Fat-Tree Topology, VLAN Segmentation, and InfiniBand vs RoCEHPC network design technical guide: why HPC networking is a separate discipline, fat-tree topology with three layers, four VLAN segments, …
- InfiniBand Technical Guide: Fabric Architecture, Subnet Manager, Installation, and TroubleshootingInfiniBand fabric architecture technical guide: why InfiniBand (RDMA, OS bypass), HDR200 vs NDR400 comparison table, fat-tree topology, …
Let's Build Future Together.
