SAS Learns Token Importance for Sparse Attention

Simple Attention Sparsification (SAS) replaces layer-wise distillation with end-to-end training to align token selection with language modeling performance. Standard Transformers compute dense attention across an entire sequence, which forces computational cost to scale quadratically with sequence length. Previous sparse attention methods relied on secondary selection heads to prune tokens based on heuristics or distilled rankings. These methods prioritize matching a pre-calculated dense attention distribution rather than the task-specific language modeling loss. SAS changes this by integrating the selector’s scores directly into the attention logit calculation.

The Logit Injection Mechanism

Existing trainable sparse attention methods typically rely on a two-step process where a selector identifies tokens and a hard selection layer picks the winners. This creates a non-differentiable mask that breaks the gradient path, preventing the selector from receiving feedback on how its choices impacted the final loss. SAS removes this barrier by injecting continuous scores from the selector into the attention mechanism before the softmax operation. If Q and K are the query and key vectors, the attention score for a token i is modified by adding the selector output S_i to the logit.

By keeping these scores in the logit space, the gradient flows from the loss function through the softmax operation back into the selector during backpropagation. Unlike distillation, where the selector targets a teacher's distribution, SAS optimizes the selector for next-token prediction perplexity directly. The model learns to assign high values to tokens that, when attended to, reduce the perplexity of the next-token prediction. The log-space bias is the primary mechanism here; it scales the attention weights before normalization, directly weighting the influence of specific tokens on the attention map.

Differentiable Budget Constraints

To enforce a strict token budget, SAS uses normalized softmax gates to approximate the selection process. Hard thresholding is non-differentiable, making it impossible for the gradient to flow through the choice of which tokens to keep. The normalized softmax approach creates a continuous probability distribution over the tokens. This allows the selector to be trained via backpropagation, effectively learning the rank of tokens by their importance to the cross-entropy loss function without the catastrophic failure of a hard cutoff.

Implementation and Efficiency

To handle long sequences, SAS uses a custom Triton kernel that integrates these selection scores directly into the FlashAttention loop. Rather than materializing a sparse matrix in memory, the kernel processes tokens in blocks. The selector calculates importance scores for these blocks, and the kernel performs a fused operation where it only computes attention for the blocks that exceed a dynamic threshold. By maintaining this computation entirely within the GPU's fast SRAM, the implementation avoids the overhead of loading irrelevant tokens back into HBM.

For a practical example, consider a 32k context window. With dense attention, the memory cost is 32k squared, which quickly exhausts GPU memory. SAS allows the model to selectively ignore blocks of tokens that the gradient feedback has identified as irrelevant to the current prediction. In practice, this means at extreme lengths, you only compute attention for a fraction of the total tokens, such as keeping the top 25% of tokens by importance. This maintains performance while reducing the quadratic memory growth that otherwise limits sequence length.

What remains unclear is the exact behavior of SAS across diverse domains. While the method optimizes for general perplexity, it is not yet known if the learned importance scores generalize well to tasks that require global context like long-form summarization or complex codebases. The dynamic thresholding mechanism also relies on the selector's confidence, and how this interacts with the 'lost' information from dropped tokens in high-variability tasks is a variable factor that researchers are still tuning.