WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models

By Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, Dong Yu. Annual Meeting of the Association for Computational Linguistics, 2024

410 citations recorded by the source corpus.

The rapid advancement of large language models (LLMs) has led to a new era marked by the development of autonomous applications in real-world scenarios, which drives innovation in creating advanced web agents. Existing web agents typically only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. To bridge this gap, we introduce WebVoyager, an innovative Large Multimodal Model (LMM) powered web agent that can complete user instructions end-to-end by interacting with real-world websites. Moreover, we establish a new benchmark by compiling real-world tasks from 15 popular websites and introduce an automatic evaluation protocol leveraging multimodal understanding abilities of GPT-4V to evaluate open-ended web agents. We show that WebVoyager achieves a 59.1% task success rate on our benchmark, significantly surpassing the performance of both GPT-4 (All Tools) and the WebVoyager (text-only) setups, underscoring the exceptional capability of WebVoyager. The proposed automatic evaluation metric achieves 85.3% agreement with human judgment, indicating its effectiveness in providing reliable and accurate assessments of web agents.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/19261c6ad20c6c1e5585a8afcb88196173cbc8a6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (2024, neutral): “We report grounding accuracy on ScreenSpot (Agent Setting), with GPT-4o as the planner.”
Large Language Models for Information Retrieval: A Survey (2023, neutral): “Specifically, the WebGPT model employs special tokens to execute actions such as querying, scrolling through rankings, and quoting references on search engines.”
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, neutral): “…and further push the boundary of next-level artificial general intelligence, spanning from embodied intelligence (Geng et al., 2023), autonomous driving (Wen et al., 2023; Cao et al., 2023; Yang et al., 2023d) to graphical user interfaces (GUI) agents (He et al., 2024; Yang et al., 2023f).”
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (2025, neutral): “…[34]; • Visual Grounding : RefCOCO-avg (val) [25], TreeBench [54], Ref-L4 [6]; • GUI Agents : OSWorld [62], Android World [38], WebVoyager Some [18], Webquest-QA [56]; • Coding : Design2Code [43], Flame-React-Eval [1]; • Spatial Reco & Reasoning : OminiSpatial [24], CV-Bench [51], ERQA [49],…”
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (2024, background): “The enthusiasm for building such agents is evident by the growing number of computer control agent systems [41, 9, 70, 24, 22, 16] and code repositories [13, 1, 2, 57].”