Workload-Conditioned Health Residual Graphs for GPU Fleets
1. Research premise
Static thresholds and workload-blind anomaly detectors can confuse normal phase changes with hardware degradation. The note therefore treats workload confounding, observability confounding, and the gap between passive evidence and active diagnosis as separate problems that must be represented explicitly.
2. Proposed representation
Each accelerator is represented through a typed health residual graph containing accelerators, HBM stacks, interconnects, hosts, cooling domains, workload phases, telemetry sources, RAS events, and diagnostic procedures. Nodes and edges retain residual magnitude, confidence, persistence, exposure denominator, timestamp, model version, and provenance rather than collapsing evidence into one anomaly score.
3. Residual construction
Observed behavior is compared against four distinct reference bases: workload-conditioned expectation, the device's own history, an equivalently configured peer cohort, and optional physics-constrained expectations. The multi-reference structure is intended to help distinguish fleet-wide or environmental effects from device-specific deterioration.
4. Exposure-normalized health
Reliability evidence is normalized by physical exposure rather than wall-clock time. The note sketches subsystem embodiments for thermal behavior, power-performance efficiency, HBM ECC and row-remap evidence, PCIe/NVLink health, control-plane state, and observability continuity.
5. Diagnostic loop
The graph state generates candidate actions such as increasing telemetry frequency, requesting additional fields, running DCGM or vendor diagnostics, replaying a controlled workload, comparing a peer, checking cooling or power sensors, collecting logs, or draining a device. Actions are ranked by expected information gain, disruption, duration, safety, operational constraints, and cost of delay.
6. Status
This is a conceptual research note shared for open discussion and collaboration. It does not claim completed empirical validation. The proposed next step is an offline DCGM replay prototype evaluated against historical fleet telemetry, incident tickets, and RMA outcomes.