JarvisGUI and the Limits of Cross-Device Agents

JarvisGUI is a new benchmark for evaluating GUI agents across Android, Windows, and Ubuntu. Most existing benchmarks test agents on static tasks where the environment remains fixed. JarvisGUI assesses agents on workflows that require state transfer, context persistence, and long-horizon dependencies across different operating systems. This focus exposes performance gaps that are often masked by single-device testing.

Traditional benchmarks present agents with tasks within one interface, such as clicking a button in a browser. JarvisGUI frames tasks as multi-step workflows. It uses a lightweight type system to define data objects that must persist across steps. When an agent moves between platforms, this system expects the model to maintain continuity of the information being processed. The benchmark is designed to determine whether models can handle the transitions that real-world work necessitates.

For those building agents, the benchmark findings suggest that high scores on single-device tests do not translate to capability in multi-step, multi-platform environments. Agents often struggle because they lack sufficient state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management. They rely on the visual context of the immediate screen, which causes them to lose intent when the interface changes. When an agent moves from one operating system to another, it struggles to connect the sequence of events into one logical workflow.

To bridge the gap between platforms, an agent would need a way to maintain an abstract representation of its current progress that persists even when the visual UI completely changes. A model currently relies on its ability to perceive visual pixels, but these pixel changes between an Android app and a desktop application are so drastic that the agent loses the thread of its intent. If an agent had a persistent state-tracking layer, it would need to store intermediate data in a format independent of the specific OS being used. Such a layer would effectively act as a clipboard that survives an interface change, requiring the model to map its internal data state to the new UI structure regardless of the visual appearance of that UI.

What remains unknown is whether current vision-language models can be fine-tuned to bridge this divide or if the architecture requires a fundamental change to how they manage memory. If you are building tools for cross-device workflows, the current evidence suggests that you are navigating an unproven space where models frequently fail to link disconnected visual events. Future work in this area will likely focus on whether we can build models that treat the environment as a persistent, typed state rather than a series of visual inputs.