Discrete Beckmann Transport Models for Token Generation

Discrete Beckmann Transport Models (DBTM) replace the multi-step, stochastic denoising of traditional diffusion models with an autonomous flow. Where diffusion relies on a student model mimicking a teacher's complex trajectory, DBTM formulates token generation as a transport problem. The model learns a vector field that defines a direct, deterministic path from an initial location in the ambient space to a target vertex representing a specific token. This allows the system to converge on an output by following this field until it reaches a fixed point.

The Mechanism of Transport

The architecture calculates a velocity vector for a given input, which guides the current probability distribution toward the intended target. Unlike diffusion, which requires conditioning on a noise schedule, this vector field is time-independent. The model does not have a single fixed field for all outcomes. Instead, the input embedding acts as a condition that modulates the neural network, effectively switching or morphing the geometry of the field to ensure the target vertex is the local sink for that specific request.

During training, the model is fed an initial distribution and the ground truth token. The training objective minimizes the residual of the conservation equation. This equation enforces a physical constraint: the total probability mass, which must sum to 1, is preserved as it is relocated through the ambient space. By forcing the predicted velocity to align with the gradient pointing toward the target vertex, the network learns to map any initial state onto the correct coordinate. Because the model learns this geometric mapping, it can execute the transformation in a single forward pass or through iterative refinement.

Solving the Transport Problem

If the network predicts a velocity vector that would cause a distribution to drift toward an incorrect token, the residual of the conservation equation spikes. This high loss signals the optimizer to adjust the neural network weights, ensuring that paths converge precisely on the Dirac delta distribution of the target token.

Consider an example of choosing between 'cat' and 'dog'. If the initial distribution is a 50/50 split, the system sits at a midpoint in the ambient space. If the input context is 'The feline meowed', the input embedding adjusts the neural network to produce a vector field where the 'cat' vertex exerts a stronger attractive force. The mass shifts along a path directed toward the 'cat' corner, while the 'dog' vertex becomes unreachable. In larger vocabularies, the space becomes a complex, high-dimensional object with tens of thousands of vertices. The difficulty is path collision: as the vocabulary grows, the model must learn the geometry with enough precision to prevent the vector field from forcing the distribution toward a neighboring, incorrect vertex.

The model currently fails to resolve long-context ambiguity where the vector field is forced to shift mid-trajectory. Future research must address how these fields remain stable when the input is not clearly resolved, as the current architecture assumes a clear destination that may not exist until late in the generation process.