Android in the Wild: A Large-Scale Dataset for Android Device Control

By Christopher Rawles, Alice Li, Daniel Rodríguez, Oriana Riva, Timothy P. Lillicrap. Neural Information Processing Systems, 2023

358 citations recorded by the source corpus.

There is a growing interest in device-control systems that can interpret human natural language instructions and execute them on a digital device by directly controlling its user interface. We present a dataset for device-control research, Android in the Wild (AITW), which is orders of magnitude larger than current datasets. The dataset contains human demonstrations of device interactions, including the screens and actions, and corresponding natural language instructions. It consists of 715k episodes spanning 30k unique instructions, four versions of Android (v10-13),and eight device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It contains multi-step tasks that require semantic understanding of language and visual context. This dataset poses a new challenge: actions available through the user interface must be inferred from their visual appearance. And, instead of simple UI element-based actions, the action space consists of precise gestures (e.g., horizontal scrolls to operate carousel widgets). We organize our dataset to encourage robustness analysis of device-control systems, i.e., how well a system performs in the presence of new task descriptions, new applications, or new platform versions. We develop two agents and report performance across the dataset. The dataset is available at https://github.com/google-research/google-research/tree/master/android_in_the_wild.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/060e0df71620f1844839f6a993dfa5fb8e4c3bf6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Ming Ding
2. Jie Tang
3. Wenyi Hong
4. Yuxiao Dong
5. Zhiyong Wu
6. Jiazheng Xu
7. Weihan Wang
8. Timothy P. Lillicrap

Citations in the corpus

• CogAgent: A Visual Language Model for GUI Agents (2023, positive): “Instead of relying exclusively on textual inputs such as HTML [28] or OCR results [31], VLM-based agents directly perceive visual GUI signals.”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…(en) [240], WebSight (en) [122], Widget-Caption (en) [136], RICOSCA (en) [55], Seeclick (en) [37], ScreenQA (en) [92], AMEX (en) [22], AITW (en) [198], Odyssey (en) [168], GUI UIBert (en) [12], AndroidControl (en) [135], Mind2Web (en) [57], OmniACT (en) [106], WaveUI (en) [4] Type: Multi-Image…”
• SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (2024, methodology): “Notably, with an equivalent amount of training data (2.8K), SeeClick outperformed the previous offline sota WebGUM, et al., 2023) and the latest GPT-4V (Yan et al., 2023); (2) our own trained LVLM baseline Qwen-VL.”
• AppAgent: Multimodal Agents as Smartphone Users (2023, neutral): “The Android in the Wild (AITW) [28] dataset contains 715,000 operation instances across 30,000 tasks.”
• Group-in-Group Policy Optimization for LLM Agent Training (2025, positive): “…such as DQN [34] to train LLM agents in text-based games [35] and later research [36, 37, 38, 39, 40] started to employ value-based methods, such PPO [41] and AWR [42], in more diverse and interactive agent scenarios including Android device control [43], embodied ALFWorld [5], and card games [44].”
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (2024, neutral): “…et al., 2017; Humphreys et al., 2022) to diverse and complex real-world environments, including the web (Deng et al., 2023; Zhou et al., 2024; Yao et al., 2022), desktop (Xie et al., 2024; Wu et al., 2024) and mobile operating systems (Rawles et al., 2023; Yan et al., 2023; Rawles et al., 2024).”
• AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (2024, background): “This kind of measurement is not representative of real-world performance [41, 70] because it does not reflect that there are usually multiple correct paths to solve a task, environments may behave non-deterministically (e.g., the environment freezes), and agents can dynamically learn from mistakes…”
• Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UI (2025, unknown): “…expectation } { evaluation_result } </point> """ 4 Open Source Infotainment Validation Dataset: AutomotiveUI-Bench-4K Given that multiple datasets [25, 33, 32, 6, 38, 1, 8] already exist in the fields of desktop, mobile, and web, we identify a domain gap in scientific research for benchmarking…”