Refining Model Reasoning with Negative Self-Distillation
Negative Self-Distillation shifts model optimization from imitating correct outputs to systematically avoiding the patterns found in flawed reasoning.
The Mechanism of Negative Divergence
Standard On-Policy Self-Distillation trains a student model to mirror the output of a teacher that is already constrained to correct ground-truth paths. This approach effectively forces the student to mimic a specific solution, which often prioritizes the output format over the reasoning process. Negative Self-Distillation replaces this imitation objective with a divergence task. Instead of learning to follow a correct path, the model is trained to actively avoid the probability distribution generated by a 'negative teacher' that is fine-tuned on intentionally corrupted reasoning traces to mimic common logic failures.
The optimization goal is to maximize the Kullback–Leibler divergence between the student model and the negative teacher. By pushing its probability mass away from sequences where the negative teacher excels, the student learns to avoid specific failure modes like premature conclusions or non-sequiturs. This is fundamentally different from standard imitation where the model converges toward a single pre-defined target. Here, the model maps out the contours of its own potential mistakes.
| Feature | OPSD (Standard) | NSD (New) |
|---|---|---|
| Primary Driver | Imitation of ground-truth | Divergence from flaws |
| Supervision | Required | Label-free |
| Reasoning Focus | Output alignment | Process self-correction |
Isolating Logic Errors
Applying a uniform penalty for incorrect reasoning risks degrading the model's fundamental linguistic fluency. To prevent this, the system employs a gating mechanism that filters the divergence signal by comparing the log-probabilities of tokens across the student model and the negative teacher. The model calculates the difference in likelihood for every generated token. If a token appears with high probability in both models, the gate assumes it belongs to standard syntax or common phrasing and effectively ignores it. If a token shows a significant spike in likelihood for the negative teacher while remaining low in the student's output, it is identified as a reasoning-critical error.
The model updates by minimizing the loss against the negative teacher only for these gated error tokens. This reduces the probability mass assigned to sequences that correlate with the negative teacher's logic failures. This approach is surgical. Because the gating logic specifically targets tokens associated with the negative teacher's errors, the model retains its base capability to generate standard, fluent text while learning to steer away from its idiosyncratic failure patterns. This provides a way to improve logic performance without modifying the core weights of the original base language model.
Consider an illustrative case where a model is used to generate proofs. It might frequently introduce logical leaps where it skips a necessary mathematical step while using perfectly correct notation. The negative teacher, trained on proof fragments containing these specific leaps, assigns high probability to the faulty bridge tokens. The student model detects this divergence in the log-probabilities of those bridge tokens, marks them as high-error, and lowers the probability mass it assigns to them in future iterations. The surrounding, correct mathematical syntax remains untouched by the gate, allowing the student to retain its fluency while correcting its logical trajectory.