Layer-Selective Unlearning in LLMs

FOM-UL is a layer-selective framework that unlearns targeted information by updating specific transformer layers instead of applying global weight adjustments. Conventional unlearning methods, such as Gradient Ascent or Negative Preference Optimization, modify all parameters in the network simultaneously. This methodology treats the transformer as a monolithic block, which risks erasing unrelated knowledge and often fails to permanently purge information during model compression.

The Mechanism of Selective Erasure

Transformers encode information hierarchically. Early layers typically capture syntactic structures and low-level features, while deeper layers integrate these into complex, high-level factual and relational knowledge. Because specific concepts often reside in particular regions of the network, modifying every weight is unnecessary. FOM-UL identifies these high-signal regions by calculating a significance score for every layer. It does this by computing the ratio of the gradient norm on the forget set—the data you want removed—to the gradient norm on the retain set—the data you want to keep. By ranking layers based on this ratio, the framework applies updates only to the layers where the targeted information is most concentrated. This concentration of force ensures that the majority of the model’s weights remain untouched, which preserves the network's original reasoning capabilities and linguistic patterns.

Why Quantization Matters for Unlearning

Standard unlearning techniques often fail under 8-bit or 4-bit quantization because the small, diffuse weight adjustments are rounded away during compression. When you perform unlearning globally, the changes to individual parameters are often negligible and spread thin. When you later apply quantization, these minor adjustments are frequently discarded or subsumed by the rounding process, causing the 'forgotten' content to leak back into the model's outputs. FOM-UL circumvents this by applying larger magnitude weight updates to the specific, identified layers. Because the updates are concentrated within a few layers, the changes remain statistically significant enough to withstand the precision loss inherent in down-casting a model.

Consider an unlearning task where you need to remove a specific set of training examples. If you spread a small total update across 50 layers, the delta per weight is often below the threshold of significance for the quantization function. If you isolate the top five most significant layers, the delta per weight is ten times larger. This larger delta acts like a deeper cut into the model's latent representation of that data. When the quantization process rounds the values, it acts as a filter that might erase a small change, but it is less likely to erase a large, focused shift in the layer's activation patterns.

Implementation Trade-offs

Calculating these layer-wise significance scores requires extra computational overhead during the preparation phase. While this adds prep time, it removes the need for post-quantization re-validation. The method demonstrates that by restricting updates to layers that focus on the specific forget-set data, you avoid injecting noise into the layers responsible for general language understanding. Empirical results show that this method outperforms established baselines like SURE, ReLearn, and LUNAR, maintaining utility closer to the original, vanilla model.

Despite these performance gains, FOM-UL provides no formal mathematical guarantee of erasure. It is an empirical approach that suppresses memorization to a degree sufficient for common safety and privacy use cases, but it does not formally prove that the knowledge has been scrubbed from the weights. What remains unknown is how this approach scales to extremely large models with hundreds of layers. As models grow, layer interaction becomes more non-linear, and it is unclear if the significance score will remain as predictive of retention and forgetting as it is in current architectures.