InfiniBand Cabling: Choosing Between DAC, AOC, Transceivers and MTP/MPO

In HPC network projects the surprise line item — in both budget and schedule — is usually not the switch but the cabling. Switch and adapter selection is relatively clear-cut; on the cable side, distance, connector type, module body, breakout planning and structured cabling decisions all interlock. A wrongly chosen module can cost weeks after the boxes have arrived.

This article works through InfiniBand cabling decisions in order.

Three options: DAC, AOC, transceiver + fibre

DAC is the default for in-rack links: passive copper, close to zero power draw, cheapest, and the least failure-prone option. Its limit is reach — usable length shrinks as the rate rises.

AOC is a single assembly with optical modules embedded at both ends. It is practical for medium rack-to-rack distances, with no separate module ordering or compatibility hassle. The downsides: if one end fails the whole cable is replaced, and the length is fixed.

Transceiver + fibre trunk is what large deployments choose. Modules and cables are managed separately; combined with structured cabling, a rack-to-rack link becomes infrastructure rather than a cable. On failure only the module is replaced.

A practical rule: DAC inside the rack, transceiver + trunk between racks, AOC for the few special links in between.

Connector: QSFP56 or OSFP?

The connector changes with each generation, and this is where mistakes are most common in mixed deployments.

QSFP56 and OSFP are physically different and do not mate. When adding NDR switches to an HDR cluster, plan the cable type at the transition points from the start.

Finned vs flat-top OSFP: the detail that gets missed

OSFP modules come in two body types, and they are not interchangeable:

• Finned body: used on the switch side. Switch port density is high, so the module needs its own heat-sink surface.
• Flat top: used on the server adapter (HCA) side. There is no room for fins in the adapter cage.

The two ends of the same link need different body types. If a quote does not make this distinction, half the modules will not fit when the boxes arrive. This is the single most common supply error we encounter.

Breakout: splitting one port into two

High-rate ports can usually be split into two lower-rate links. This affects port economics directly.

HDR100. On the QM8700/QM8790, 200G QSFP56 ports split into two separate two-lane 100G ports. With ConnectX-6 adapters, one switch reaches 80 nodes instead of 40.

NDR twin-port. On the QM9700/QM9790, 32 physical OSFP cages carry 64 logical 400G links using twin-port modules. Switch count and rack space for the same spine capacity drop.

Two points matter when using breakout: the cable type must match the split configuration, and the port-split setting must be applied on the switch during commissioning. Buying the cable and forgetting the setting leaves the port showing as a single link.

MTP/MPO: structured cabling between racks

Rather than pulling dozens of individual fibres between racks, using multi-core MTP/MPO trunks and cassettes is the common and correct approach.

The structure: a high-count trunk is run between racks, an MTP/LC cassette is fitted into a patch panel at each end, and distribution inside the rack is done with short LC patch cords.

The gains:

• Cable-path density drops. One trunk replaces dozens of separate fibres.
• Commissioning speeds up. The trunk is pulled once; later node additions stay inside the rack.
• Airflow is preserved. Cable bundles do not block the rear of the rack.
• Documentation becomes possible. Cassette ports can be labelled and mapped.

The thing to watch is polarity (Type A/B/C). The polarity scheme of trunk, cassette and patch cords must be consistent; otherwise links do not come up, and this is one of the most frustrating faults to diagnose on site.

Distance planning: do not order before measuring

Cable lengths get calculated on paper from rack spacing, but real length is always more: cable path, vertical drops, service loop and in-rack routing all add up.

A practical approach:

1. Produce the rack layout; decide which node connects to which leaf switch.
2. Measure by physically following the cable path — not straight-line distance.
3. Add a service allowance, but do not overdo it: excess cable piles up behind the rack and disturbs airflow.
4. Round lengths to standard steps (0.5 / 1 / 1.5 / 2 / 3 m). Stock management and spares get simpler this way.

Spares: the 3-5% rule

Optical modules and cables are consumables. Keeping 3-5% of the total count per type as spares is common practice in production. A node dropping out of the cluster for the length of a procurement cycle because of one module failure costs far more than the spares.

Mevasis cabling and supply

Cabling decisions cannot be separated from switch and adapter selection: finned OSFP on the switch side, flat top on the HCA side, DAC or trunk depending on distance — all of it has to be consistent in one list.

As an authorised dealer, Mevasis supplies DAC and AOC cables, transceiver modules and MTP/MPO structured cabling as a single compatibility-verified quote. Share your rack layout and we will produce the length list and the polarity scheme — get in touch.

Frequently Asked Questions

Can I use a flat-top module on the switch side? No. High-density switch ports need a finned body for cooling; even if a flat-top module fits physically, it is not thermally supported. Specify the module type for each end separately before ordering.

How long can a DAC cable be? It depends on the rate, and shrinks as the rate rises. In practice in-rack links (a few metres) are DAC and links leaving the rack are planned as optical. In borderline cases, confirm the manufacturer’s maximum length for that rate.

How much does active optical cabling raise power draw? Noticeably. The Q3400-RA, for example, draws a typical 2,900 W with passive cables and a maximum of 7,000 W in a fully active-optical configuration. Do not finalise the rack power budget before the cable type is settled.

Is using third-party “compatible” modules risky? It works technically, but the responsibility boundary blurs on failure and firmware compatibility can cause trouble across version upgrades. In production clusters, sourcing switch and modules from the same supply chain shortens diagnosis time.