OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

By Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang and 1 more. arXiv.org, 2024

338 citations recorded by the source corpus.

Existing efforts in building GUI agents heavily rely on the availability of robust commercial Vision-Language Models (VLMs) such as GPT-4o and GeminiProVision. Practitioners are often reluctant to use open-source VLMs due to their significant performance lag compared to their closed-source counterparts, particularly in GUI grounding and Out-Of-Distribution (OOD) scenarios. To facilitate future research in this area, we developed OS-Atlas - a foundational GUI action model that excels at GUI grounding and OOD agentic tasks through innovations in both data and modeling. We have invested significant engineering effort in developing an open-source toolkit for synthesizing GUI grounding data across multiple platforms, including Windows, Linux, MacOS, Android, and the web. Leveraging this toolkit, we are releasing the largest open-source cross-platform GUI grounding corpus to date, which contains over 13 million GUI elements. This dataset, combined with innovations in model training, provides a solid foundation for OS-Atlas to understand GUI screenshots and generalize to unseen interfaces. Through extensive evaluation across six benchmarks spanning three different platforms (mobile, desktop, and web), OS-Atlas demonstrates significant performance improvements over previous state-of-the-art models. Our evaluation also uncovers valuable insights into continuously improving and scaling the agentic capabilities of open-source VLMs.

Subfields: Artificial Intelligence, Social Psychology

View Paper: https://www.semanticscholar.org/paper/9ceadd197cea2cd1b45e12983d528daf2b24efaa

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wenhai Wang
2. Xizhou Zhu
3. Lewei Lu
4. Weijie Su
5. Zhiqi Huang
6. Yulun Du
7. Chenzhuang Du
8. Zhilin Yang

Citations in the corpus

• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, positive): “In For the more challenging ScreenSpot-V2 [130] benchmark, InternVL3 exhibits strong scaling behavior: InternVL3-72B achieves 90.9%, outperforming UI-TARS-72B (90.3%).”
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, neutral): “MMLU-Pro [61], GAOKAO [177], IFEval [185]; (4) Agentic Tasks : SGP-Bench [102], ScreenSpot [16], ScreenSpot-v2 [150], OSWorld-G [156], VSI-Bench [161], ERQA [121], SpaCE-10 [38].”
• Kimi-VL Technical Report (2025, positive): “ScreenSpot V2 (Zhiyong Wu et al. 2024) is an enhanced version of the ScreenSpot (K. Cheng et al. 2024) benchmark, which focuses on evaluating the performance of GUI grounding models across multiple platforms, including web, desktop, and mobile interfaces.”
• Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UI (2025, unknown): “…} { evaluation_result } </point> """ 4 Open Source Infotainment Validation Dataset: AutomotiveUI-Bench-4K Given that multiple datasets [25, 33, 32, 6, 38, 1, 8] already exist in the fields of desktop, mobile, and web, we identify a domain gap in scientific research for benchmarking LAMs…”