Automating Embodied Benchmark Synthesis

Embodied-BenchForge replaces manual, multi-stage benchmark construction with an automated, agentic workflow that verifies intermediate artifacts during synthesis. Researchers developed this framework to address cumulative error propagation in embodied task creation, where a single faulty dependency—such as a misplaced asset or an unreachable target—invalidates an entire evaluation dataset. By treating benchmark construction as a directed acyclic graph, the system ensures each component is validated against a set of constraints before the pipeline proceeds.

The Problem with Linear Synthesis

Existing approaches to building embodied benchmarks typically rely on linear, open-loop pipelines. You start with a scene, generate tasks, create ground truth, and export the result. If the task environment configuration drifts, the error cascades. Because these systems rarely verify intermediate artifacts like pathing constraints or object placement, you often discover the defect only after the full benchmark has been rendered and deployed. Embodied-BenchForge modularizes the scene into discrete nodes. Each node represents a distinct skill or environment state, and each is verified against a contract before the next stage proceeds.

Construction and Graph Management

Each contract acts as a gatekeeper using specific predicate checks on the physical state of the simulation. These predicates operate as programmatic geometric filters. For example, a reachability check performs a raycast between an agent start location and a target coordinate within the simulation physics engine to verify that the path is unobstructed by colliders. If the raycast returns a collision signal, the predicate fails.

When a component fails its predicate contract, the system triggers local re-execution or upstream rollback. The system maintains a mapping between the failed predicate and the associated scene metadata, identifying the specific parameter responsible for the violation. Rather than discarding the entire workspace, the system iterates on that local component by re-processing the metadata fields until the contract is satisfied. The relationship between the violation log and the scene metadata is managed by a graph orchestrator that tracks the unique ID of every asset. When a failure occurs, the orchestrator points to the exact coordinate ID or scene graph node that failed, ensuring that only the relevant segment of the environment is updated for the next iteration.

Comparison of Construction Paradigms

Implementation and Limitations

The system automates the creation of six benchmarks for the Offline EQA Track and one interactive benchmark containing 220 tasks for the Interactive Embodied Track. The current architecture requires every task to map to the predefined skill library, which acts as the foundation for the verification predicates. The system is restricted to these defined skills and has not been tested on open-world generation or novel environment layouts outside of the initial library. The system cannot generate environments requiring non-standard object interactions or physics properties that are not explicitly defined within its foundational skill library. It remains to be seen how the orchestrator scales when the complexity of these dependencies grows beyond the current limit of several hundred tasks, specifically regarding the computational cost of continuous verification in high-fidelity simulation environments.