Alpha-Split Privacy for Federated Speech Models
Alpha-split mitigates privacy budget collapse in federated speech models by decoupling sensitivity-based clipping for encoders and decoders.
Federated speech-LLMs rely on differential privacy to obscure user data, but standard implementations often trigger a budget collapse that ruins model performance. This occurs when training processes use a single, global privacy budget to clip gradient updates across vastly different model components that operate at different sensitivity scales. The alpha-split method prevents this by applying independent clipping thresholds for the encoder and decoder while maintaining the original (ε, δ) privacy guarantee for the entire model.
The Problem of Uniform Clipping
In standard federated learning, you clip gradients to ensure no single device's contribution exceeds a specific sensitivity threshold. Per-layer differential privacy methods typically allocate the privacy budget to layers proportional to their parameter count. This approach assumes that gradient sensitivity is uniform throughout the network.
The acoustic encoder processes high-frequency raw audio data and generates update norms that are often orders of magnitude smaller than those from the language decoder. When you apply a uniform clipping threshold, the system is forced to choose a single value for the entire model. If you set the threshold too high to accommodate the decoder, the encoder's gradients may exceed the desired sensitivity bound, risking exposure. If you set it low to protect the encoder, the decoder's updates are clipped beyond their optimal range, degrading utility and forcing an injection of noise that causes the word error rate to spike significantly.
Independent Thresholds and Composition
Alpha-split resolves this by treating the encoder and decoder as distinct entities with their own clipping thresholds. Because the privacy budget is additive under independent parallel composition, this method ensures the total privacy loss across the combined model remains bounded by the original (ε, δ) target. Instead of forcing a single, suboptimal threshold, the mechanism allows the system to set a dedicated clipping boundary Ce for the encoder and Cd for the decoder.
This approach relies on the fact that the total privacy cost of the system is the sum of the privacy loss for each component. By allowing the encoder and decoder to operate under different clipping constraints, the method avoids the noise inflation that results from applying a decoder-level threshold to the smaller gradients of the encoder.
| Feature | Uniform Per-Layer DP | Alpha-Split DP |
|---|---|---|
| Budget Pool | Global | Independent pairs |
| Clipping Logic | Proportional to size | Calibrated to sensitivity |
| Protection against gradient-inversion attacks | Weak | 4.47x tighter |
| Decoder Utility | High noise overhead | Restored to baseline |
Impact on Edge Hardware
On edge hardware, the noise overhead of standard DP often prevents the use of deep architectures. Alpha-split adds a minor noise overhead of about 2.6% to the model. By dropping the encoder threshold Ce to match its true, lower distribution, the noise variance injected into the audio features is reduced by a factor of 4.47 compared to uniform methods. This allows the model to maintain parity with non-private baselines at the same total (ε, δ) budget, whereas uniform clipping would require a significantly larger privacy budget to achieve the same word error rate.
What remains uncertain is the scalability of this calibration approach to models with more than two distinct functional components. The alpha-split mechanism is currently optimized for the encoder-decoder architecture of speech models; whether it provides similar gains for multi-modal architectures—such as those adding vision or depth-sensing encoders—is not yet clear. The overhead of managing more than two distinct clipping thresholds may also introduce new, currently unquantified bottlenecks in on-device gradient calculation.