RoCEv2 and Ultra Ethernet: A Real Alternative to InfiniBand?

“InfiniBand or Ethernet?” is answered on a separate comparison page. This article covers the step after that: if you have chosen the Ethernet side, how do you make it fit for HPC — and will Ultra Ethernet genuinely change the picture?

The short version: plain Ethernet is not enough for HPC. What makes it enough is RoCEv2 and the lossless transport underneath it — a layer that does more harm than good when misconfigured. Below we cover how RoCEv2 works, why PFC and ECN settings are critical, and which weaknesses Ultra Ethernet targets.

Why HPC Networking Is a Specialized Topic

Millisecond-level network latency is irrelevant in desktop or enterprise IT environments. In a 1,024-core MPI simulation, however, 100 µs of latency per MPI call accumulates into hours of lost compute time.

Two metrics drive HPC network selection:

• Latency: Time for a message to travel from one node to another. Microseconds and below matter.
• Bandwidth: Volume of data transferable per second, measured in Gb/s.
• MPI collective performance: The network topology’s role in operations like Allreduce and Broadcast.

InfiniBand: Core Concepts

InfiniBand was designed in 1999 specifically for HPC and data center applications. Its key differentiator is RDMA (Remote Direct Memory Access): data is copied directly to remote memory, bypassing the CPU and dramatically reducing latency.

Generations and Speed Classes

New deployments in 2026 should target HDR200 or NDR400.

InfiniBand’s Distinguishing Features

RDMA: Direct access to remote node memory without CPU interrupt or OS involvement — the primary source of InfiniBand’s latency advantage.

Zero-Copy: Data transfers directly from application memory to the network without intermediate buffering.

Kernel Bypass: Network operations are handled directly by the HCA (Host Channel Adapter) hardware, bypassing the OS kernel.

High-Speed Ethernet: RoCE and DPDK

Standard Ethernet protocol stacks are too slow for HPC. Two technologies bring Ethernet closer to HPC requirements:

RoCE (RDMA over Converged Ethernet)

Provides InfiniBand’s RDMA advantage over Ethernet infrastructure. Two versions:

• RoCE v1: Layer 2 only; same subnet
• RoCE v2: Layer 3, routable; requires lossless Ethernet with PFC and ECN

RoCE v2 requires careful switch configuration and Priority Flow Control (PFC) to prevent performance-destroying packet loss.

Performance Comparison

Single-Message Latency

MPI Allreduce at Scale (1,024 cores, 1 MB message)

Network selection affects total simulation time by 20–40% for MPI-intensive workloads.

Fat-Tree Topology

The most common HPC topology. Every node reaches every other node in the same number of hops, eliminating congestion (oversubscription).

         Core Switches
        /      |       \
    Spine    Spine    Spine
   /    \   /    \   /    \
 Leaf  Leaf Leaf  Leaf Leaf Leaf
  |     |    |    |    |    |
 N1    N2   N3   N4   N5   N6

1:1 (non-blocking): All nodes can communicate simultaneously at full speed. Higher cost; for large installations.

2:1 oversubscribed: Core bandwidth halved; 30–40% lower cost. Sufficient for most HPC workloads.

Cost Comparison

64-Node HPC Cluster

InfiniBand costs 5–15× more than Ethernet, but for MPI-intensive workloads the performance advantage often justifies the premium.

Ultra Ethernet: which weaknesses does it target?

RoCEv2 has three well-known weaknesses in the field, and the Ultra Ethernet Consortium (UEC) targets exactly those.

1. Mandatory in-order delivery

RoCEv2 assumes packets arrive in order. That assumption makes multi-path routing unusable in practice: traffic funnels onto a single path and congests it while other links sit idle.

Ultra Ethernet permits out-of-order delivery and leaves reordering to hardware at the endpoint. Packets can then spread freely across all paths and the fabric’s real capacity becomes usable.

2. Dependence on PFC

Lossless Ethernet today is built on PFC (Priority Flow Control). When a buffer fills, PFC sends a “pause” to the other side. It works, but it has two well-documented side effects:

• Head-of-line blocking. A paused flow also stalls unrelated flows queued behind it.
• Congestion spreading. The pause propagates upstream, and congestion widens to switches that were never its source. In the worst case a PFC storm locks up the entire fabric.

UEC targets a loss-tolerant transport instead: treating packet loss as a normal event rather than a disaster, and compensating with fast retransmission.

3. Per-connection state

In classic RDMA, state is kept for every connection pair. On a fabric where thousands of GPUs perform collectives, that becomes a serious load on adapter memory. UEC aims at a lighter connection model to make large-scale fabrics easier to scale.

What to do today

An honest assessment: Ultra Ethernet products are only now reaching the field. They are not comparable in maturity to InfiniBand’s decade-plus of production history, mature diagnostics and well-understood behaviour.

• Buying within 12-24 months? Base the decision on hardware you can buy today. Note Ultra Ethernet as a future option, not a purchase justification.
• Planning five years out? Account for the direction Ethernet is moving, and build the cabling and rack plan to carry higher port rates later.
• Deploying RoCEv2 today? Put the PFC and ECN settings into the commissioning report. These are not set-and-forget; they need re-measuring as the workload profile changes.

GPUDirect RDMA

NVIDIA GPUDirect RDMA exposes GPU memory directly to the InfiniBand network adapter — no CPU involvement, no system memory staging.

Traditional path: GPU → CPU (pinned memory) → NIC → Network
GPUDirect RDMA:   GPU → NIC → Network  (CPU bypassed)

In distributed deep learning (NCCL AllReduce), this feature reduces communication overhead by 20–30%.

Mevasis Network Design Services

Mevasis provides consulting and deployment services for HPC cluster network design and InfiniBand installation. Contact our technical team for NVIDIA Quantum-2 InfiniBand and high-speed Ethernet solutions.

Frequently Asked Questions

Can InfiniBand and Ethernet coexist in the same cluster? Yes. A dual-network architecture is common: Ethernet for management and storage traffic, InfiniBand for MPI communication. This provides good cost-performance balance.

Is RoCE difficult to configure? RoCE v2 requires PFC and ECN configuration, making it more complex than standard Ethernet. Incorrect configuration leads to packet loss and severe performance degradation. Expert configuration is recommended.

Is InfiniBand necessary for a small deployment (8 nodes)? Generally no. At 8 nodes or fewer, 25GbE or RoCE 100GbE offers better cost-performance balance. InfiniBand advantage becomes clear at 32+ nodes and MPI-intensive workloads.