HPC Monitoring Service — Infrastructure Observability and Alerting

Enterprise HPC infrastructure goes unnoticed while it runs cleanly; the moment something fails, every minute counts. A proactive monitoring system catches problems before users feel them.

Monitoring Stack Components

Prometheus — Metric Collection

Prometheus is a time-series collection and query system. Several exporters run across an HPC cluster:

# Prometheus scrape configuration
scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['compute[01-32]:9100']  # node_exporter

  - job_name: 'slurm'
    static_configs:
      - targets: ['mgmt01:9341']          # slurm_exporter

  - job_name: 'dcgm'
    static_configs:
      - targets: ['gpu[01-04]:9400']      # DCGM exporter

node_exporter — Hardware Metrics

Runs on every compute node:

• CPU utilisation, per core
• Memory and swap usage
• Disk I/O (read/write MB/s, IOPS)
• Network traffic (InfiniBand and Ethernet)
• System load average
• Hardware sensors (CPU temperature, fan speed)

DCGM Exporter — GPU Health Monitoring

NVIDIA Data Center GPU Manager converts GPU metrics into Prometheus format:

• GPU utilisation (%)
• GPU memory usage and bandwidth
• GPU power draw (watts)
• GPU temperature and thermal throttling state
• PCIe and NVLink error counters
• Memory ECC error counts (correctable / uncorrectable)

SLURM Exporter — Queue Metrics

• Running / pending / suspended job counts
• Partition utilisation
• Resource usage by user and project
• Average queue wait time

Grafana — Visualisation

Pre-built HPC dashboards:

1. Cluster overview: the state of every node on one screen
2. GPU health: four panels for GPU temperature, utilisation and a power map
3. Storage performance: I/O throughput and latency
4. Job accounting: CPU and GPU hours by project and user

Alertmanager — Intelligent Alerting

# Example critical alert
- alert: GPUHighTemp
  expr: dcgm_gpu_temp > 85
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "GPU temperature crossed the critical threshold"

Alerts are delivered by email, SMS or to a Slack channel.

To build your monitoring infrastructure or improve an existing system, get in touch.