Scaling Robot Control Through Shared Trajectory Diffusion

Existing robot policies typically rely on autoregressive training, where a model outputs a single action based on a current observation. Dynin-Omni shifts this by using a masked-diffusion backbone that treats language, vision, motor actions, and goal states as discrete tokens within an omnimodal space. Instead of separate planning and execution modules, the model predicts the trajectory—a sequence of actions—and the corresponding visual changes in the environment simultaneously.

The core of this improvement is the joint prediction objective. By training the model to denoise both the robot's next motor commands and the future visual state, the system ensures that the selected action sequence aligns with a target outcome. The model computes the loss across both modalities during training, forcing the learned internal representation to capture the causal link between motor movement and environmental change. For example, in a reach-to-grasp task, the model performs joint refinement: it evaluates trajectory candidates against predicted visual outcomes to select paths that successfully reach the target rather than intersecting with obstacles.

Decoding and Parallelism

Standard diffusion models are slow because they are iterative. To generate an action, the model typically performs many sequential denoising passes. Dynin-Omni changes this by treating the sequence of future tokens as a block. Rather than denoising one action at a time, the decoder predicts a block of trajectory tokens simultaneously. This parallel structure is made possible by a non-causal attention mask across the trajectory block. While standard transformers use a lower-triangular causal mask to ensure tokens only attend to the past, this implementation allows bidirectional attention between tokens within the same block.

The model resolves the dependency issue through an iterative refinement process on the block itself. Because every step in the sequence is constrained by the initial denoising pass and subsequent refinement iterations, the model maintains temporal coherence. The bidirectional attention allows tokens within a block to influence each other during the refinement phase, effectively collapsing the iterative loop of traditional diffusion into an efficient block-based inference pass. This approach allows the system to generate longer horizons of control in fewer total passes.

The model is pre-trained on 1.33 million trajectories across 48 datasets from the Open X-Embodiment collection. The parallel implementation provides up to 29.2x faster model-side action decoding compared to the base implementation. This allows the model to handle larger chunks of the motion path in a single pass. The efficiency gains are primarily in the inference throughput, as the block-based decoding requires fewer total passes through the model than autoregressive alternatives.

Implementation Hurdles

The model achieved an average 78.4% success rate across four manipulation conditions on the Franka Research 3 robot. These figures rely on a fixed Stage-2 step budget for adaptation. The primary challenge for a developer is balancing the inference parameters during the refinement loop. While the model is capable of joint prediction, the integration of these two modalities into a singular output requires careful configuration of the guidance scale. A developer must calibrate the influence of the visual goal prediction relative to the action sequence tokens to ensure the generated trajectory remains physically viable. This balance is not automatically enforced by the architecture, and improper calibration can lead to trajectories that appear visually aligned with a goal but are kinematically impossible for the hardware to execute. We still lack clarity on the precise interaction between visual goal guidance and the noise schedule at extreme scales, which remains a key area for future testing as these models are deployed on more complex manipulators.