Boosting Tabular Model Inference with FP8 Attention

The researchers' Triton kernel achieves a speedup of up to 1.7x for tabular foundation models like TabPFN-v3 and TabICLv2 by quantizing attention queries, keys, and values to FP8. This optimization targets the attention arithmetic directly, shifting from 16-bit operations to 8-bit to reclaim compute cycles during the transformer forward pass.

Why Lower Precision Speeds Up Matrix Operations

Tabular models often process distinct feature-row inputs, making the volume of attention arithmetic the primary performance bottleneck. Converting queries, keys, and values to FP8 reduces the memory footprint of these operands before calculation. The hardware executes this by processing the 8-bit data through native Tensor Core paths, designed to perform multiply-accumulate operations on reduced-precision inputs. This increases throughput because the silicon spends less time moving data from memory and more time executing arithmetic. Unlike LLMs, where the KV cache size often dictates performance, tabular models see gains here by reducing the data transfer overhead relative to the total compute time.

Aligning Quantization Error Through Training

Moving from 16-bit to 8-bit precision risks accuracy loss if the quantization noise is not accounted for. To maintain performance, the researchers utilize quantization-aware training. The training process simulates the noise introduced by FP8 precision during the forward pass. The loss function incorporates this noise, training weights to compensate for FP8 rounding errors. Crucially, the model is trained so that it remains robust to the specific quantization distribution it will encounter during inference; accuracy drops drastically if the quantization error in test rows is not aligned with that of the training rows. To handle the non-differentiable nature of rounding during this process, the implementation uses straight-through estimators, which allow gradients to flow through the rounding operation as if it were an identity function during backpropagation.

Real-World Implementation

To understand the practical impact, consider a model predicting credit risk based on a batch of 512 tabular rows. In a standard 16-bit implementation, the GPU spends substantial time moving FP16 tensors into the arithmetic units. By switching to FP8, you effectively double the available bandwidth for these tensors, allowing the kernel to saturate the Tensor Cores more effectively. However, this is not a universal performance boost. It requires NVIDIA Hopper or Blackwell GPUs to leverage native FP8 instructions. If you attempt to deploy this on earlier architectures, the lack of native FP8 support forces the GPU to fall back on emulation paths, which are slower than standard 16-bit math. Currently, it remains unknown how well this approach generalizes to extremely sparse tabular feature sets, where the distribution of activation values may be too skewed to benefit from standard FP8 dynamic range limits.