GPT-4V(ision) is a Generalist Web Agent, if Grounded

By Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, Yu Su. International Conference on Machine Learning, 2024

582 citations recorded by the source corpus.

The recent development on large multimodal models (LMMs), especially GPT-4V(ision) and Gemini, has been quickly expanding the capability boundaries of multimodal models beyond traditional tasks like image captioning and visual question answering. In this work, we explore the potential of LMMs like GPT-4V as a generalist web agent that can follow natural language instructions to complete tasks on any given website. We propose SEEACT, a generalist web agent that harnesses the power of LMMs for integrated visual understanding and acting on the web. We evaluate on the recent MIND2WEB benchmark. In addition to standard offline evaluation on cached websites, we enable a new online evaluation setting by developing a tool that allows running web agents on live websites. We show that GPT-4V presents a great potential for web agents -- it can successfully complete 51.1 of the tasks on live websites if we manually ground its textual plans into actions on the websites. This substantially outperforms text-only LLMs like GPT-4 or smaller models (FLAN-T5 and BLIP-2) specifically fine-tuned for web agents. However, grounding still remains a major challenge. Existing LMM grounding strategies like set-of-mark prompting turns out to be not effective for web agents, and the best grounding strategy we develop in this paper leverages both the HTML structure and visuals. Yet, there is still a substantial gap with oracle grounding, leaving ample room for further improvement. All code, data, and evaluation tools are available at https://github.com/OSU-NLP-Group/SeeAct.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c844694387a89a477e7a8bbf918171cdc3b85672

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Graham Neubig
2. Fei Huang
3. Yu Su
4. Shuyan Zhou
5. Jing Yu Koh
6. Timothy P. Lillicrap
7. Zhiyong Wu
8. Ji Zhang

Citations in the corpus

• OS-ATLAS: A Foundation Action Model for Generalist GUI Agents (2024, background): “Many current agents make decisions based on textual descriptions of the environments, such as HTML and accessibility trees, which is often lengthy (Zheng et al., 2024a), noisy (Cheng et al., 2024; WebAIM, 2024), and hard to acquire in practice.”
• An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models (2024, unknown): “From describing the given picture to navigating the internet Zheng et al. (2024), using smartphones Wang et al. (2024) and making decisions in the real world Driess et al. (2023); Chen et al. (2024), large language models with vision abilities are reshaping how we interact with AI systems, which…”
• Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception (2024, methodology): “Zheng et al. [2024] points out that although GPT-4V can generate effective operations, it struggles to accurately locate the positions of these operations on the screen.”
• Group-in-Group Policy Optimization for LLM Agent Training (2025, unknown): “For instance, they now power embodied assistants that navigate simulated homes [5, 6], mobile and web navigators that plan multi-step browsing sessions [7, 8, 9], and autonomous explorers in interactive games [10, 11].”
• WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024, background): “Concur-rently with our work, SeeAct (Zheng et al., 2024) also leverages Large Multimodal Models (LMMs) for integrated visual understanding and actions on websites.”
• TravelPlanner: A Benchmark for Real-World Planning with Language Agents (2024, background): “Previous research has demonstrated that language agents can effectively decompose tasks and engage in step-by-step reasoning, leading to significant improvements (Wei et al., 2022; Yuan et al., 2023; Zheng et al., 2024).”
• VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024, background): “(Zheng et al., 2024) is contemporaneous work which performs action grounding to identify appropriate HTML elements for enabling VLM agents to execute actions on the web.”
• AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (2024, methodology): “This kind of measurement is not representative of real-world performance [41, 70] because it does not reflect that there are usually multiple correct paths to solve a task, environments may behave non-deterministically (e.g., the environment freezes), and agents can dynamically learn from mistakes…”
• SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (2024, background): “However, constrained by the inherent limitation of LLMs to only process text and the complexity of GUIs, converting interfaces into structured text is cumbersome and challenging to standardize across various GUI environments.”
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (2024, unknown): “Zheng et al. (2024) have clearly demonstrated the necessity of visual perception for web agents, so we mainly compare with zero-shot methods that use MLLMs as planners and omit text-only LLMs. Zheng et al. (2024) have also identified grounding as the main challenge and proposed several grounding…”