ASTRIL-MPC: Robot Control with Semantic Guidance

ASTRIL-MPC integrates large language models into the control loop of tracked robots to navigate complex, unstructured terrain. It pairs an LLM, which provides semantic context, with a Non-linear Model Predictive Control (NMPC) solver that manages the physics of movement. This architecture separates the slow-moving semantic reasoning, which operates on visual scene classification, from the high-frequency control loop, which must maintain stable motion at speeds far beyond the inference limits of an LLM.

The Mechanism of Adaptive Control

Unlike traditional controllers that rely on rigid analytical kinematics, ASTRIL-MPC uses a neural network to predict short-horizon task-state increments. The network processes the terrain by embedding raw point cloud data into a local latent representation, allowing the system to map geometric features like surface gradients and step heights into a localized coordinate prediction. By anticipating how tracks engage with these features, the robot can calculate a motion plan before physically traversing the surface.

To adapt to varying terrains, the LLM provides dynamic tuning for the NMPC solver. The NMPC formulation explicitly accounts for contact variables between the robot tracks and the ground. The LLM classifies the visual environment—such as recognizing mud or loose rubble—and proposes bounded updates to the cost weights within the solver's optimization objective. By adjusting these weights, the solver forces the optimization process to favor specific trajectory profiles, such as prioritizing trajectory smoothness over pure speed, which directly influences the velocity profile and track interaction sequences commanded to the robot's hardware.

Constraining Semantic Input

LLMs can produce unpredictable outputs, so the system implements a safety-checked interface that treats the LLM as an untrusted signal source. This ensures that semantic suggestions remain within the bounds of physical reality.

• Range Clipping. The interface forces every weight update into pre-defined numerical bounds to prevent the controller from reaching unstable configurations.
• Rate Limiting. The system restricts how quickly cost weights can evolve, which prevents the robot from oscillating between competing control strategies as the terrain changes.
• Consistency Checks. The interface validates that the proposed updates maintain physical viability, ensuring that calculated torque values remain within the robot's joint specifications.

While this framework improves performance in diverse environments, it remains an open question how the system will scale to truly unseen terrains where the LLM might hallucinate non-existent ground properties. The reliability of this system depends on the robustness of the safety-checked interface, as the underlying NMPC solver can only be as stable as the constraints provided to it. Future work will need to determine if this hierarchical structure holds up when the robot encounters dynamic, high-speed obstacles that require reaction times faster than the current semantic reasoning layer can provide.