OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

By Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, T. Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei and 7 more. Neural Information Processing Systems, 2024

919 citations recorded by the source corpus.

Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity. However, existing benchmarks either lack an interactive environment or are limited to environments specific to certain applications or domains, failing to reflect the diverse and complex nature of real-world computer use, thereby limiting the scope of tasks and agent scalability. To address this issue, we introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks that involve arbitrary applications. Building upon OSWorld, we create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld reveals significant deficiencies in their ability to serve as computer assistants. While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge. Comprehensive analysis using OSWorld provides valuable insights for developing multimodal generalist agents that were not possible with previous benchmarks. Our code, environment, baseline models, and data are publicly available at https://os-world.github.io.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ff3e4f7c2481fb6df539f02be5945235101cbc19

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (2025, neutral): “From solving scientific problems [67; 32; 34] to developing autonomous agents [21; 62; 43], the demands on VLMs have far surpassed simple visual content perception [31], with an increasing emphasis on advanced reasoning abilities.”
AppAgent: Multimodal Agents as Smartphone Users (2023, neutral): “Recent systems like Mind2Web [8], WebArena [53], WorkArena [9], and VisualWebArena [18] have leveraged LLMs to operate and interact with web pages, while others, such as OSWorld [41], extend similar capabilities to desktop PCs.”
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (2024, methodology): “Researchers have applied this idea to software engineering [20, 61, 15], robotics [34], vision [29, 57], and GUI navigation [54, 23, 53].”
InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, positive): “MMLU-Pro [61], GAOKAO [177], IFEval [185]; (4) Agentic Tasks : SGP-Bench [102], ScreenSpot [16], ScreenSpot-v2 [150], OSWorld-G [156], VSI-Bench [161], ERQA [121], SpaCE-10 [38].”
Group-in-Group Policy Optimization for LLM Agent Training (2025, neutral): “Early works typically relied on leveraging pre-trained, frozen models through carefully designed prompting methods (like ReAct [28] and Reflexion [29]), enhanced memory and retrieval systems [11, 30], and integration with external tools [31, 32, 33].”
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (2024, neutral): “(Yan et al., 2023; Kim et al., 2023; Wang et al., 2024a; Zheng et al., 2024; Xie et al., 2024).”
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (2024, methodology): “A similar result was obtained in recent work in the context of computer agents for desktop applications [59].”
Qwen3-VL Technical Report (2025, positive): “For online evaluations, Qwen3-VL 32B scores 41 on OSWorld and 63.7 on AndroidWorld, which surpasses the current foundation VLMs. Qwen3-VL demonstrates exceptionally strong planning, decision-making, and reflection abilities as a GUI agent.”
Neurosymbolic Repo-level Code Localization (2026, unknown): “End-to-end issue resolution benchmarks, such as SWE-Bench [12] and its extensions, assess full issue-solving capabilities [2–4, 6, 8, 9, 12, 18–20, 31, 33, 37], primarily focus on bug-fix tasks and limited programming languages.”
LocAgent: Graph-Guided LLM Agents for Code Localization (2025, unknown): “These successful examples are then used to refine the same 32B model further, reinforcing effective reasoning patterns through this self-improvement loop.”