Decoupling Transformer Inference with SQD Disaggregation

The SQD (SubQuadratic Disaggregation) framework optimizes transformer inference by splitting attention operations based on their mathematical complexity. Instead of forcing a monolithic GPU to handle both dense matrix multiplication and sparse attention lookups, the system assigns these tasks to pipelines tailored to their specific computational requirements. This approach acknowledges that quadratic attention and subquadratic, or sparse, indexing create fundamentally different demands on a processor's memory architecture.

The Problem with Monolithic Attention

Historically, inference systems treat the transformer block as a single unit, keeping attention and feed-forward network calculations on the same accelerator. This architecture ignores that dense attention is compute-bound, relying on heavy matrix math, while sparse indexing requires frequent, irregular lookups into the Key-Value (KV) cache. On a standard GPU, these workloads fight for the same physical registers and memory controllers. Pressure from matrix math conflicts with the cache requirements of indexing logic, resulting in stalls where arithmetic units idle while waiting for metadata.

Disaggregating the Pipeline

SQD separates these operations by isolating sparse indexing into modules that handle lookups independently of the primary GPU compute pipeline. Rather than forcing the GPU to manage sparse metadata, the system offloads indexing logic to hardware that traverses sparse data structures directly. By delegating index traversal, the GPU avoids the overhead of managing non-contiguous memory access within its SIMT (Single Instruction, Multiple Threads) architecture.

Synchronization between these streams relies on a hardware-level credit-based flow control protocol. The GPU grants 'credits' to the indexing module for specific memory regions, which represent pre-allocated buffer space in the indexing module's local SRAM. Before the indexing module can initiate a lookup, it must have a credit available for that memory segment; this prevents it from overrunning the GPU's command buffer and causing a memory deadlock. When the GPU executes a compute kernel, it monitors a status register updated by the indexing module. If the indexing module has not yet released the data or consumed the necessary credits, the compute kernel triggers a targeted micro-stall. This deterministic dependency ensures the GPU never reads stale results while keeping the arithmetic units utilized on other thread blocks.

Latency and Fabric Constraints

Data moves between the GPU and the indexing module via a dedicated direct-path fabric. This fabric uses a fixed-width bus optimized for command signaling. While this isolation prevents sparse traffic from saturating the main memory bus, it introduces a physical bottleneck. Unlike a GPU's internal memory hierarchy, which uses deep buffering and caching to hide latency, this fabric is exposed to the raw latency of the interconnection.

In a multi-tenant cloud environment, this is the primary point of failure. If an unrelated tenant generates high-priority traffic on the shared fabric, the resulting congestion induces non-deterministic delays. Because the SQD protocol requires data to be present before the compute kernel can proceed, any delay on the fabric that exceeds the synchronization threshold propagates back to the GPU as a compute stall. The system effectively trades the efficiency of disaggregation for a strict reliance on predictable interconnect latency.

In an adjusted 8xB200 heterogeneous system proxy, this scheduling strategy yields a 56% improvement in tokens per joule for the Gemma 4 31B model compared to standard GPU-only baselines. Analytical models for future hardware like the Rubin and LPX architectures project a 3.6x throughput advantage when comparing this disaggregated approach against standard attention-FFN baseline methods. The reliability of these gains hinges on the interconnect’s ability to manage contention without forcing the synchronization protocol into a wait-state.