InfiniBand Subnet Manager: Fabric Management with OpenSM and UFM

No packet moves on an InfiniBand fabric without a Subnet Manager (SM). The switches are racked, the cables are run, the adapters are installed — but if no SM is running, ports sit in Initializing and not a single MPI message gets through.

This component has no Ethernet equivalent, and it is the part of an InfiniBand deployment most often misunderstood and most often left without redundancy.

What exactly does the Subnet Manager do?

The SM scans the fabric and then:

1. Discovers. Finds every switch, adapter (HCA) and link between them, and builds the topology map.
2. Addresses. Assigns a LID (Local Identifier) to each port. InfiniBand routing works over these LIDs.
3. Computes routes. Calculates the forwarding table for every switch and writes it to them.
4. Activates ports. Moves ports to Active; data traffic only starts after this.
5. Keeps watching. Recomputes when the topology changes — a cable pulled, a switch failure, a node added.

In other words the SM is not a monitoring tool but the control plane of the fabric.

Managed vs unmanaged switches

Product lists show two models with the same port count: QM8700 and QM8790, or QM9700 and QM9790. The hardware is almost identical; the difference is precisely here.

The common and correct approach: choose the first switch in the fabric — and a second for redundancy — as managed, and buy the ones added while scaling as unmanaged. If an SM is already running in the fabric, every switch does not need its own, and this lowers the cost of expansion noticeably.

A fabric built entirely from unmanaged switches is also possible — in that case the SM runs as OpenSM on a server inside the cluster.

OpenSM: a subnet manager on a server

OpenSM is InfiniBand’s open-source subnet manager and ships with MLNX_OFED / DOCA-OFED. It typically runs on a login or management node.

# Is it installed, is the fabric visible?
ibstat                  # local HCA port state
ibnetdiscover | head    # fabric topology
sminfo                  # which SM is active?

sminfo is the first place to look in production: it shows which node is the SM and at what priority.

Redundancy: a single SM is not a production setup

The operational mistake we see most often is running one OpenSM on one node. When that node reboots, existing connections survive for a while — but the moment anything in the topology changes, the fabric is left unmanaged.

A correct setup runs SM on at least two nodes. InfiniBand supports this natively: SMs negotiate on priority, the highest becomes master and the other standby. When the master goes down, the standby takes over.

# /etc/opensm/opensm.conf — on the master node
priority 15

# on the standby node
priority 10

Giving different priorities matters; with equal priority the election falls back to GUID and which node becomes master turns unpredictable.

Choosing a routing engine

OpenSM’s default routing algorithm is not the best choice for every topology. On fat-tree deployments, ftree or updn makes a clear difference:

# on a fat-tree topology
routing_engine ftree

The wrong routing engine causes congestion on certain links even in a physically non-blocking fabric. During commissioning you should run cross-traffic measurements with ib_write_bw and confirm the expected bisection bandwidth is reached.

Partitions (pkeys): dividing the fabric

In shared clusters different groups may need to be isolated from each other’s traffic. InfiniBand does this with partitions: each has a pkey, and only ports carrying the same pkey communicate.

# /etc/opensm/partitions.conf
Default=0x7fff, ipoib, defmember=full : ALL;
research=0x8001, ipoib, defmember=full : 0x0002c90300abcdef, 0x0002c90300abcdf0;

A typical use: separate partitions for compute, storage and management traffic. In multi-tenant deployments this provides both isolation and traffic separation.

UFM: when scale grows

OpenSM does its job but offers no monitoring or management interface. NVIDIA UFM (Unified Fabric Manager) sits on top of that: central topology view, port counters, error-threshold alerts, telemetry and automatic event logging.

A rough rule: for deployments of a few dozen nodes, OpenSM plus ibdiagnet is enough. At hundreds of nodes — especially in shared environments with an SLA — the visibility UFM provides reduces operational load noticeably.

Diagnosis: when something goes wrong

ibdiagnet                    # fabric-wide health scan
ibhosts                      # visible HCAs
ibswitches                   # visible switches
iblinkinfo                   # link rates and states
perfquery -a                 # port error counters

ibdiagnet is the first command to run in production; it gathers credit loss, symbol errors and rate mismatches into a single report.

Three common findings:

A port runs slower than expected. Cable or module rate mismatch, or a dirty/loose connector. Compare expected against actual rate in iblinkinfo.

Rising symbol errors. A physical-layer issue — cable, module or port. Clear the counters and read them again after a set interval to measure the error rate; a single accumulated number is misleading.

Two master SMs on the fabric. Usually appears after a network partition. Check with sminfo and make sure priority settings are consistent.

Mevasis fabric commissioning services

Subnet manager setup, redundant configuration, routing engine selection and partition planning are part of our HPC infrastructure deployment service. Commissioning is handed over together with an ibdiagnet report and cross-traffic bandwidth measurements.

We also help plan the managed/unmanaged switch mix — see InfiniBand switch models or get in touch.

Frequently Asked Questions

I bought unmanaged switches — where should the subnet manager run? On a server inside the cluster, usually the login or management node, as OpenSM. In production we recommend running it on at least two nodes with different priorities.

If both switches are managed, do two SMs run? Yes, and that is the desired state. One becomes master, the other standby; the standby takes over if the master fails. What matters is setting different priorities.

Do running jobs break if the SM restarts? Established connections are usually unaffected, because the data plane works independently of the SM. But if the topology changes while no SM is running — a cable pulled, say — the fabric cannot recover on its own.

Does using partitions hurt performance? No. The pkey check happens in hardware and adds no measurable latency.