RetroThinker Improves Reasoning in Moshi Speech Models
RetroThinker allows speech-based LLMs to perform retrospective reasoning corrections during a single generation pass without increasing latency.
RetroThinker is a framework designed for the Moshi model that allows it to revise its reasoning during speech generation. Traditional speech models rely on static Chain-of-Thought sequences, which creates a rigid path that collapses when early logic fails. By using supervised fine-tuning on retrospective data and length-based direct preference optimization, this approach enables the model to issue verbal corrections as it generates tokens. This addresses the challenge where models typically struggle to pivot once an incorrect logical trajectory has begun.
The Mechanism of Verbal Correction
When a model generates output in a streaming fashion, it usually commits to a sequence immediately. RetroThinker alters this by training the model to prioritize output paths that reach a correct answer, even if the reasoning sequence involves a mid-stream shift. The model does not physically overwrite previously emitted audio; instead, it learns to generate correction tokens, such as 'wait' or 'actually', that signal the correction of a previous error within the same stream.
During training, the model uses length-based direct preference optimization to penalize unnecessarily long or incorrect reasoning sequences. It essentially learns the statistical probability of needing a reversal token when it encounters an error in its own early logic. This makes the correction a native part of the generation flow rather than a response to an external trigger. The model is trained to favor shorter, accurate paths, which naturally constrains the amount of time it spends correcting itself versus generating the final answer.
| Feature | Standard Moshi | RetroThinker |
|---|---|---|
| Reasoning Path | Fixed | Dynamic |
| GSM8K Accuracy | Baseline | +11% Absolute Gain |
| Latency Impact | Negligible | Negligible |
Consider an agent calculating a 20% discount on a $100 purchase. If the model incorrectly estimates $10 during early token generation, it relies on this learned post-training behavior to immediately generate a follow-up sequence—saying, 'Wait, that is not correct, the discount is $20.' The user hears the correction as part of the live audio stream because the model has incorporated the revision into the ongoing token sequence. This keeps the agent in a state of continuous speech without requiring the system to pause for re-prompting or server-side computation.
Currently, this implementation is validated only for tasks with verifiable outputs like GSM8K. It has not been tested for subjective tasks or creative writing, where a 'correct' path is not easily defined by a numeric result. Because the correction mechanism relies on the model identifying a mismatch between its current tokens and the ground truth in its training data, its efficacy in open-ended or non-verifiable domains remains unknown. Future iterations will need to determine if this training objective can be generalized to tasks that lack a discrete, verifiable final state.