NVIDIA SHARP: Aggregating Collective MPI Operations Inside the Network

However fast the GPUs get in distributed deep learning training, gradients still have to be aggregated at every step. That aggregation — MPI_Allreduce, or ncclAllReduce on the NCCL side — becomes the dominant term in total time as scale grows. SHARP takes part of that work off the servers and moves it inside the switch.

The problem: why doesn’t all-reduce scale?

In a classic all-reduce, each node sends its data to neighbours, combines what arrives and redistributes the result. Even with efficient algorithms like ring-allreduce, the total volume moved grows with node count and each node sees the same data more than once.

The consequence: a communication share that is negligible at 8 nodes can eat half the step time at 256. Swapping in faster GPUs does not fix this — quite the opposite, as compute speeds up the proportional share of communication grows.

What SHARP does

SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) organises the reduction as a tree built on the fabric topology. Data from leaf nodes is aggregated on the leaf switch, partial results move up a level, the final result forms at the root and is distributed back down the same tree.

The critical difference: the aggregation happens inside the switch ASIC, at wire speed. Instead of circulating repeatedly between endpoints, the data goes up once and comes down once.

In practice:

• Less data moves. Fewer bytes cross the fabric to produce the same result.
• Hop count is bounded. Reduction time grows with tree depth — logarithmically — rather than linearly with node count.
• CPU and GPU are freed. Because the reduction arithmetic happens in the network, compute resources stay on the actual work.

Generations: SHARPv1 to SHARPv4

The important break is SHARPv2: it moved beyond accelerating barriers on small messages to aggregating large vectors at wire speed. Gradient synchronisation falls exactly into that profile, which is why SHARP started being associated with machine learning at this version.

SHARPv3’s multi-tree support matters in shared clusters: different jobs build their own reduction trees on the same fabric and do not wait on each other.

What do you need?

SHARP is not a single setting but a stack of requirements:

1. A supporting switch. The NVIDIA Quantum, Quantum-2 or Quantum-X800 families — QM8700/QM8790 on HDR, QM9700/QM9790 on NDR, Q3400-RA/Q3200-RA on XDR.
2. An Aggregation Manager. The service that builds and manages SHARP trees. It runs on a managed switch or on a server in the cluster.
3. A healthy Subnet Manager. Trees are built from the real fabric topology, so the SM has to be consistent and healthy.
4. Application-side support. NCCL and HPC-X/Open MPI can use SHARP behind the scenes. No application code change is needed; the work happens in the library layer.

Enabling and verifying

The rough order: bring up the Aggregation Manager service, scan the fabric and build the trees, then configure the collective library to use SHARP.

On the NCCL side the relevant plugin must be installed and enabled. The most practical way to verify is to turn on NCCL debug output and look at which algorithm the collective selects — if SHARP is active, it shows up there.

# Which algorithm is NCCL choosing?
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,COLL
mpirun -np 16 ./all_reduce_perf -b 8 -e 2G -f 2 -g 1

all_reduce_perf from nccl-tests is the standard way to measure the SHARP gain. Run it twice: once with SHARP off, once on. Make the comparison on the same set of nodes and over the same message-size range, otherwise the result is misleading.

How much does it gain? The honest answer

SHARP’s gain is not a fixed percentage. Three factors decide it:

Scale. At 8 nodes the difference is usually noise. The gain becomes pronounced as node count rises — that is SHARP’s whole reason to exist.

Message size. On very small messages latency dominates and the gain is limited. On very large messages bandwidth dominates. The clearest benefit is usually in the wide band between them.

Communication share. If communication is 10% of step time, halving it shortens total time by at most 5%. Amdahl’s law applies here too.

So treat claims like “SHARP speeds things up by 30%” with caution. The right approach is an A/B measurement with nccl-tests on your own workload, and a decision based on your own numbers.

Common problems

SHARP does not appear active. The most common cause is the Aggregation Manager not running or not being able to scan the fabric. Second most common: the NCCL plugin is not installed.

Trees fail to build. If the topology does not match what SHARP expects, no tree forms. This shows up in mixed-rate or irregular fat-trees.

Works for some jobs, not others. In shared clusters the number of concurrent trees is limited, and that limit is tighter before SHARPv3.

No difference in measurement. Often not a real problem: your workload’s communication share is low, or the scale is below the level where SHARP makes a difference.

Mevasis fabric design and commissioning

SHARP is built on top of the right topology and a healthy subnet manager; it does not arrive automatically with the switch purchase. Beyond supplying InfiniBand switches, Mevasis covers SHARP-aware topology planning, Aggregation Manager setup and a before/after measurement report with nccl-tests as part of commissioning.

Share your cluster scale and workload profile and we will assess together whether SHARP produces a meaningful gain in your case — get in touch.

Frequently Asked Questions

Do I need to change my application code for SHARP? No. NCCL and HPC-X/Open MPI use SHARP transparently at the collective layer. The application keeps calling MPI_Allreduce or ncclAllReduce.

Does SHARP work with unmanaged switches (QM8790, QM9790)? The switch hardware supports it, but an Aggregation Manager has to run somewhere. With no managed switch, that service runs on a server in the cluster.

Can SHARP and adaptive routing be used together? Yes, they operate at different layers. Adaptive routing spreads point-to-point traffic; SHARP aggregates collectives on a tree.

Does SHARP help single-node multi-GPU training? No. Intra-node communication goes over NVLink and never reaches the fabric. SHARP applies to inter-node collective traffic.