GPU Utilization Is a Broken Metric for LLM Inference
High utilization metrics on H100 GPUs often mask deep inefficiencies in how matrix multiplication fragments are filled during low-batch token generation.
Nvidia H100 inference workloads frequently report high Streaming Multiprocessor (SM) utilization while delivering unexpectedly low token throughput. This contradiction emerges from an architectural mismatch between dense matrix math and the reality of token-by-token decoding. Profiling vLLM with modern kernels on Hopper GPUs reveals that traditional utilization counters collapse distinct hardware bottlenecks into a single percentage. This obscures the fact that during decode, where models generate one token at a time, the hardware is constrained by how its physical data paths handle fixed matrix fragments.
The Mechanism of Fragment Filling
The performance floor in small-batch decode is defined by how the H100 handles dense projection GEMMs. Hopper's bfloat16 GMMA path operates on matrix fragments of a fixed size, specifically 64-row blocks. This architecture exists to maximize throughput by amortizing instruction overhead across large registers, forcing the hardware to commit to a specific block size to satisfy the pipeline's structure. When you decode a single token with a batch size of one, you perform a vector-matrix multiplication that only requires the hardware to process a single row.
Because the hardware is wired to process these 64-row fragments, the Tensor Core pipeline cannot simply skip the unused portions of the block. The hardware requires a full fragment to initiate the Tensor Memory Accelerator and the subsequent math operation, meaning the pipeline is bound to the structural requirements of the block. During these cycles, the SM is reported as 'active' because it is committed to the fragment, but the execution pipes are stalled by a data dependency: the warp cannot proceed to the next instruction until the full fragment operation completes. The hardware cannot interleave or switch to other warps during this specific window because the scheduling logic is tied to the completion of the assigned fragment, effectively creating a stall where the GPU appears busy while the compute throughput remains constrained by the geometry mismatch.
Quantifying the Decode Mismatch
| Metric | Old Understanding | Reality on Hopper |
|---|---|---|
| Utilization | Single percentage across all cycles | Eight counter-validated views |
| GEMM Input | Fixed by model layer size | Constrained to fragment geometry |
| Decode Efficiency | High utilization means high performance | High utilization often masks stalls |
| Primary Bottleneck | Compute saturation | Fragment geometry and pipeline overhead |
Solving for Kernel Throughput
To move past standard telemetry, engineers use Nsight Compute counters that isolate hardware stalls. You should examine the sm__pipe_tensor_op_hmma_cycles_active counter to measure how many cycles the Tensor Core pipes are active relative to the theoretical peak. To diagnose whether kernels are struggling, compare this against sm__pipe_tensor_op_hmma_cycles_stalled_barrier and sm__pipe_tensor_op_hmma_cycles_stalled_wait_for_instruction_fetch metrics. If these stall counters are high while SM utilization is also high, you are hitting the limits of your fragment configuration.
What remains uncertain is the extent to which future compiler optimizations or kernel fusion strategies can dynamically mitigate these stalls without requiring structural changes to the workload. While we know the hardware relies on fixed-geometry fragments, the industry is still moving toward better profiling methods to visualize exactly how much 'wasted' work happens within a single 64-row block. The path forward involves shifting from aggregate utilization percentages to precise cycle-level diagnostics, allowing developers to see the difference between useful computation and architectural waiting periods. Understanding this difference is necessary for anyone attempting to optimize inference latency on modern GPU clusters.