HDR, NDR or XDR: How to Choose an InfiniBand Speed Tier

The most common mistake when picking an InfiniBand speed tier is making the decision by looking at the GPU model. “We bought Blackwell, so we need XDR” sounds reasonable but is wrong. What determines the speed tier is not how fast the nodes compute — it is how much they communicate.

This article breaks the choice between 200G HDR, 400G NDR and 800G XDR down along three concrete axes: the communication profile of the workload, port economics, and topology depth.

Three tiers in one table

The real information in that table is not in the rate column but in the port column. Each generation doubles the port rate while also raising the port count per switch. That means reaching the same node count with fewer switches and a shallower topology — and most of the cost difference is created right there.

1. Is your workload actually saturating the fabric?

The starting point is the communication intensity of the jobs running on your cluster.

Low network dependency. Embarrassingly parallel jobs, parameter sweeps, Monte Carlo simulations, inference that completes on a single node. In this profile raising the port rate produces no measurable gain; HDR is more than sufficient, and in some cases 100G Ethernet is even worth discussing.

Moderate MPI. Classic CFD, structural analysis, weather forecasting, molecular dynamics. There is regular barrier synchronisation and neighbour exchange, but message sizes are relatively small; the bottleneck is usually latency, not bandwidth. HDR’s sub-130ns switching latency is enough here. The large majority of university and enterprise HPC clusters still sit in this profile.

All-reduce heavy distributed training. This is where things change. In data-parallel training the gradients of every GPU are aggregated at each step, and the volume moved scales with model size. For a billion-parameter model, gigabytes cross the fabric every step. NDR and above show up directly in job completion time.

Training where tensor/pipeline parallelism crosses the node boundary. If a model does not fit into the GPUs of a single server, part of the intra-node NVLink traffic spills onto the fabric. That is exactly the scenario XDR targets.

A practical rule: do not upgrade the tier before you measure. Watch port utilisation on your current cluster with perfquery counters for a few weeks. If peak usage does not exceed 40% of port capacity, the next tier will bring you an invoice, not speed.

2. Port economics: an expensive port is not always expensive

List price per port rises with each generation. But total cost is determined not by port price but by the total hardware needed to reach the target bisection bandwidth.

In a non-blocking fat-tree, half of the leaf switch ports are reserved for the spine. Take a 512-node cluster:

• HDR (40 ports/switch): 20 nodes per leaf → 26 leaf switches plus a spine layer. A third level becomes likely.
• NDR (64 ports/switch): 32 nodes per leaf → 16 leaf switches, two levels are enough.
• XDR (144 ports/switch): 72 nodes per leaf → 8 leaf switches, two levels with room to spare.

As switch count drops, so do rack space, power draw, optical module count, cable count and failure surface. The optical module count in particular is missed in most quotes: every leaf-spine link means two modules, and in large builds the optics line item can exceed the switch line item.

So the comparison should be made on “the full bill of materials for the target bisection”, not on price per port.

3. Topology depth and latency

In a fat-tree, each extra level means two more switch hops on the end-to-end path. At roughly 130-150 ns per switch, fabric latency between two distant nodes in a three-level design rises noticeably over a two-level one.

MPI collectives such as MPI_Barrier and MPI_Allreduce feel this directly, because total time is the time of the slowest path. Choosing a higher-radix switch flattens the topology and narrows the latency distribution. XDR’s 144-port Q3400-RA reaches more than ten thousand adapters in a two-level fat-tree, removing the third level entirely in practice.

Mixed speed tiers: HDR and NDR on one fabric

A frequent question: can NDR switches be added to an existing HDR cluster? Yes. InfiniBand is backward compatible; a link comes up at the highest rate both ends support.

Two things need care in practice:

1. Topology planning gets harder. Traffic imbalance appears between leaves of different rates; adaptive routing compensates to a degree, but the design has to be deliberate.
2. Cable and module compatibility. QSFP56 (HDR) and OSFP (NDR/XDR) are different connectors. The cable type — and a breakout plan where needed — must be worked out at the transition points.

A common and workable approach for phased migration: attach new GPU nodes to NDR leaves, leave existing CPU nodes on HDR, and join the two islands at the spine.

HDR100: the forgotten middle step

A lesser-known capability of the QM8700 is splitting 200G ports into two separate two-lane 100G ports. With ConnectX-6 adapters, one switch connects to 80 nodes.

Where 100Gb/s per node is enough but node count is high — typically CPU-heavy university clusters — this halves the number of switches required. Ruling HDR out without putting this option on the decision table is usually premature.

Decision summary

Mevasis network design and supply

The speed tier decision is never just a switch choice: adapters, optics, cables, topology and the power plan are settled together. As an authorised dealer, Mevasis supplies InfiniBand switches, adapters, transceivers and cables as a single, compatibility-verified list.

Share your current inventory and workload profile and we will put together a measurement-based tier recommendation and migration plan — get in touch or see HPC Network Design.

Frequently Asked Questions

I bought an NDR switch but my adapters are HDR. What happens? The link comes up at 200Gb/s and works fine. The switch’s spare capacity stays unused until the adapters are upgraded. This is a deliberate choice in phased migrations.

Does XDR require active optical cables? No, passive copper DAC works at short reach. But the choice affects the power budget directly: the Q3400-RA draws a typical 2,900 W with passive cables and a maximum of 7,000 W in a fully active-optical configuration. Do not finalise the rack power plan before the cable type is settled.

How much does upgrading the tier shorten training time? The exact ratio depends on the workload. Where the network is the bottleneck in all-reduce heavy training, moving from HDR to NDR makes a clear difference; where the fabric is already idle, no measurable difference appears. Measure fabric saturation before deciding.

When does a modular chassis beat fixed switches? Roughly at the scale where the spine layer would need dozens of separate switches. At that point, building leaf-spine links across a chassis backplane removes hundreds of optical modules and cables.