CanvasAnneal: Training Diffusion Models with Teacher Traces
CanvasAnneal accelerates the training of diffusion language models by injecting teacher traces into the latent canvas, replacing blind exploration with curriculum-based guidance.
Diffusion models generate text by iteratively refining latent representations from noise. When applied to complex reasoning, these models struggle to maintain coherence because the path from initial noise to a valid, logical sequence is vast. CanvasAnneal modifies this process by introducing a curriculum-guided reinforcement learning framework. Instead of requiring the model to explore the entire latent space to discover correct reasoning chains, it periodically injects teacher-provided tokens to guide the model's trajectory.
The Mechanism of Guided Diffusion
In standard diffusion training, the model attempts to predict the noise added to a sequence across all latent positions. CanvasAnneal alters this during the reinforcement learning phase by using a teacher model to provide high-quality reasoning traces. These traces act as hard constraints, or anchors, within the latent canvas.
Technically, this is implemented by masking the loss function during the forward diffusion process. When the model performs its denoising steps, the latent tokens designated as anchors by the teacher are held constant. The loss is calculated only on the remaining, non-anchored tokens. Because self-attention layers compute representations by attending to all positions in the sequence—including the fixed anchors—the model's prediction for the free tokens becomes mathematically dependent on the context provided by the teacher. The model does not merely copy the anchors; it learns to minimize the denoising loss for the surrounding vectors subject to the state defined by the fixed tokens. By constraining the search space in this way, the model optimizes its policy based on the alignment of its free tokens with the fixed logical requirements of the teacher.
The curriculum component governs how these anchors are used over time. At the start of training, the framework injects a high density of teacher-provided tokens. As the training iterations progress, the system samples a smaller subset of the teacher's trace to include, effectively decreasing the anchor density. This schedule transitions the model from relying on external scaffolding to generating the full reasoning chain independently.
| Feature | Standard Diffusion RL | CanvasAnneal RL |
|---|---|---|
| Exploration | Global latent search | Curriculum-guided pathing |
| Initial States | Pure noise | Teacher-injected anchors |
| Anchor Schedule | None | Iterative density decay |
| Policy Optimization | Unconstrained denoising | Denoising with masked loss |
Empirical Results and Known Constraints
Researchers evaluated CanvasAnneal across benchmarks including MATH500, Countdown, and Tau2. On the MATH500 dataset, the framework reached a stable policy using approximately 40% fewer search iterations than standard diffusion-based RL approaches. The technique effectively shifts the burden of exploration from the model to the curriculum.
However, the performance of the model remains coupled to the teacher. If the teacher's reasoning trace contains logical errors, the model learns those errors as part of its target objective. While the system demonstrates that injecting anchors accelerates the rate at which a model improves its reward, it does not solve the fundamental dependency on the quality of the provided training curriculum. The degree to which this method scales to significantly harder reasoning tasks that lie outside the reach of the teacher model remains an open question in the current research.