Steering Video Models with Latent Probes

Video models often struggle to maintain consistent motion over long sequences because they prioritize learned, low-energy trajectories during inference rather than strict physical constraints. A recent study shows that the model maintains a latent representation of physical dynamics, such as velocity or spatial displacement, even when the visual output is incorrect. By applying targeted interventions to specific network layers, you can force the model to adopt accurate motion patterns that were previously masked by its default generation bias.

Translating Physical Labels into Latent Activations

The intervention involves modifying the model's residual stream with a steering vector. To derive this vector, you collect a dataset of internal model activations paired with ground-truth physical labels, such as the velocity of a moving object. You then train a linear probe, which effectively learns a mapping from the high-dimensional latent space of the model to the target scalar or vector representing the physics. The probe acts as a weight matrix that projects these physical attributes into the model's internal activation space.

When you add this computed steering vector to the residual stream, you shift the hidden state in a direction that aligns with the target physical property. Each layer of a transformer block performs a series of linear and non-linear transformations on these hidden states. By shifting the input to a later block, you adjust the values that go into the attention mechanism and the subsequent feed-forward networks. Because the attention mechanism calculates relationships based on these shifted states, the model generates outputs that reflect the new, injected physical dynamics rather than its original path.

The Threshold of Information Commitment

Evidence shows that a model's commitment to a specific trajectory is localized to depth boundaries. Before reaching a certain layer, the internal state remains malleable, allowing for low-gain interventions to dictate the motion. Beyond that point, the model's internal pathing becomes increasingly difficult to override. This happens because the model layers progressively synthesize and refine the visual concept, effectively cementing the object's trajectory into its final representation. While the latent signal for the correct motion exists even in later layers, overriding the learned bias at this stage requires high-gain interventions that often result in visual artifacts.

Finding the exact depth where the model is most receptive to guidance is necessary for successful steering. If you attempt to intervene too late, the model's internal weights have already established a stable path that resists minor adjustments. If you intervene too early, the signal may be lost or drowned out by subsequent layers that normalize the data. Identifying the sweet spot between these bounds allows you to control the motion without the distortions associated with forcing an override in late-stage layers.

Identifying What Remains Unknown

For a concrete example, consider a model generating the movement of two masses, one red and one blue, traveling at distinct velocities. In its raw form, the model might fail to distinguish their speeds, causing the masses to collapse toward a shared velocity. By probing the activations, you identify a latent direction that correlates specifically with the individual velocity of the red mass. Adding a steering vector along this direction during the forward pass forces the model to respect the higher velocity of the red object.

This technique proves that the model contains the information necessary for accurate motion, but it is not currently known whether this method scales to more complex, multi-object interactions or three-dimensional spatial environments. It is also unclear how these steering vectors transfer across different base models trained on different datasets, or if the latent directions for motion are universal features of transformer-based video generation. We currently lack a general theory for why certain layers are more receptive to these probes than others, meaning that each application currently requires individual diagnostic testing to identify the correct intervention depth.