AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

By Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, J. Waltz, G. Lau, Marybeth Fair, Alice Li, Will Bishop, Wei Li, Folawiyo Campbell-Ajala and 5 more. International Conference on Learning Representations, 2024

367 citations recorded by the source corpus.

Autonomous agents that execute human tasks by controlling computers can enhance human productivity and application accessibility. However, progress in this field will be driven by realistic and reproducible benchmarks. We present AndroidWorld, a fully functional Android environment that provides reward signals for 116 programmatic tasks across 20 real-world Android apps. Unlike existing interactive environments, which provide a static test set, AndroidWorld dynamically constructs tasks that are parameterized and expressed in natural language in unlimited ways, thus enabling testing on a much larger and more realistic suite of tasks. To ensure reproducibility, each task includes dedicated initialization, success-checking, and tear-down logic, which modifies and inspects the device's system state. We experiment with baseline agents to test AndroidWorld and provide initial results on the benchmark. Our best agent can complete 30.6% of AndroidWorld's tasks, leaving ample room for future work. Furthermore, we adapt a popular desktop web agent to work on Android, which we find to be less effective on mobile, suggesting future research is needed to achieve universal, cross-platform agents. Finally, we also conduct a robustness analysis, showing that task variations can significantly affect agent performance, demonstrating that without such testing, agent performance metrics may not fully reflect practical challenges. AndroidWorld and the experiments in this paper are available at github.com/google-research/android_world.

Subfields: Artificial Intelligence, Computer Networks and Communications, Industrial and Manufacturing Engineering

View Paper: https://www.semanticscholar.org/paper/d1b2eaf7aaebbd3d847272da04be180e35c7b68b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yu Su
2. Huan Sun
3. Rongyao Fang
4. Jingren Zhou
5. Boyuan Zheng
6. Shuai Bai
7. Boyu Gou
8. Rui Men

In Computer Networks and Communications

1. Huan Sun
2. Jingren Zhou
3. Yu Su

Citations in the corpus

• GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (2025, neutral): “…[45], MMLongBench-Doc [34]; • Visual Grounding : RefCOCO-avg (val) [25], TreeBench [54], Ref-L4 [6]; • GUI Agents : OSWorld [62], Android World [38], WebVoyager Some [18], Webquest-QA [56]; • Coding : Design2Code [43], Flame-React-Eval [1]; • Spatial Reco & Reasoning : OminiSpatial [24],…”
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (2024, positive): “Mobile: AndroidWorld.”
• Qwen3-VL Technical Report (2025, positive): “For online evaluations, Qwen3-VL 32B scores 41 on OSWorld and 63.7 on AndroidWorld, which surpasses the current foundation VLMs. Qwen3-VL demonstrates exceptionally strong planning, decision-making, and reflection abilities as a GUI agent.”