Mind2Web: Towards a Generalist Agent for the Web

By Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, Yu Su. Neural Information Processing Systems, 2023

1,276 citations recorded by the source corpus.

We introduce Mind2Web, the first dataset for developing and evaluating generalist agents for the web that can follow language instructions to complete complex tasks on any website. Existing datasets for web agents either use simulated websites or only cover a limited set of websites and tasks, thus not suitable for generalist web agents. With over 2,000 open-ended tasks collected from 137 websites spanning 31 domains and crowdsourced action sequences for the tasks, Mind2Web provides three necessary ingredients for building generalist web agents: 1) diverse domains, websites, and tasks, 2) use of real-world websites instead of simulated and simplified ones, and 3) a broad spectrum of user interaction patterns. Based on Mind2Web, we conduct an initial exploration of using large language models (LLMs) for building generalist web agents. While the raw HTML of real-world websites are often too large to be fed to LLMs, we show that first filtering it with a small LM significantly improves the effectiveness and efficiency of LLMs. Our solution demonstrates a decent level of performance, even on websites or entire domains the model has never seen before, but there is still a substantial room to improve towards truly generalizable agents. We open-source our dataset, model implementation, and trained models (https://osu-nlp-group.github.io/Mind2Web) to facilitate further research on building a generalist agent for the web.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/58f8925a8b87054ad0635a6398a7fe24935b1604

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Graham Neubig
2. Minlie Huang
3. Jie Tang
4. Mohit Bansal
5. Luke S. Zettlemoyer
6. Xiao Liu
7. Yejin Choi
8. Uri Alon

Citations in the corpus

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023, background): “, 2023) or turned to the web as an interactive setting featuring tasks that require multiple steps to solve (Yao et al., 2022; Zhou et al., 2023; Deng et al., 2023; Liu et al., 2023d).”
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (2024, positive): “Many screenshots in Multimodal-Mind2Web have giant vertical heights (e.g., 1 , 280 × 10 , 000 pixels).”
• InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024, methodology): “These agents are enabled through two primary methodologies: (1) using in-context learning to equip LLMs with the capability to utilize various tools, like ReAct (Yao et al., 2022), MindAct (Deng et al., 2023) and AutoGPT (Significant Gravitas); (2) fine-tuning LLMs with function calling examples, se”
• AgentTuning: Enabling Generalized Agent Abilities for LLMs (2023, neutral): “For incorrect output formats, we apply the BLEU metric to align the model’s response with the closest possible action from the available choices.”
• VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024, background): “Recent benchmarks in this domain introduced tasks involving actions on static internet pages (Deng et al., 2023) as well as interaction in simulated web environments (Yao et al., 2022; Zhou et al., 2024).”
• CogAgent: A Visual Language Model for GUI Agents (2023, positive): “We follow the setting of Deng et al. [10] in our experiments, and report step success rate (step SR) metric.”
• AgentBench: Evaluating LLMs as Agents (2023, background): “The evaluation involves a dual process to improve the efficiency following [19].”
• The rise and potential of large language model based agents: a survey (2025, neutral): “Currently, LLM-powered autonomous agents primarily function as human assistants, accepting tasks delegated by humans to either independently complete assignments or assist in human task completion [95,167,300,311,367,376].”
• TravelPlanner: A Benchmark for Real-World Planning with Language Agents (2024, background): “…al., 2021; Patel et al., 2021); tool-use evaluating agents’ proficiency in employing tools and reporting results (Li et al., 2023; Xu et al., 2023; Zhuang et al., 2023); and web navigation, testing agents’ ability to locate specific websites (Deng et al., 2023; Zhou et al., 2024; Liu et al., 2024).”
• A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, background): “MindAct [15] could be the most relevant work, where finetuned language model summarizes the raw HTML document into task-relevant snippets, and another language model predicts the web actions in a multichoice QA format.”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…(en) [37], ScreenQA (en) [92], AMEX (en) [22], AITW (en) [198], Odyssey (en) [168], GUI UIBert (en) [12], AndroidControl (en) [135], Mind2Web (en) [57], OmniACT (en) [106], WaveUI (en) [4] Type: Multi-Image Datasets Img-Diff (en) [101], Birds-to-Words (en) [100], Spot-the-Diff (en) [100],…”
• AppAgent: Multimodal Agents as Smartphone Users (2023, neutral): “Recent systems like Mind2Web [8], WebArena [53], WorkArena [9], and VisualWebArena [18] have leveraged LLMs to operate and interact with web pages, while others, such as OSWorld [41], extend similar capabilities to desktop PCs.”
• A survey on large language model based autonomous agents (2023, neutral): “5 The applications (left) and evaluation strategies (right) of LLM-based agents. but are highly related to this area.”
• OS-ATLAS: A Foundation Action Model for Generalist GUI Agents (2024, methodology): “Given that there are currently relatively few agent benchmarks, especially in the desktop domain, we have only utilized three datasets — AMEX (Chai et al., 2024) (mobile), AITZ (Zhang et al., 2024d) (mobile), and Mind2Web (Deng et al., 2023a) (web) — to train our model, leaving a significant number ”
• WebArena: A Realistic Web Environment for Building Autonomous Agents (2023, methodology): “Some works rely on static states, limiting agents’ explorations and functional correctness evaluation (Shi et al., 2017; Deng et al., 2023), while others simplify real-world complexities, restricting task variety (Yao et al., 2022a; Liu et al., 2018).”
• SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (2024, methodology): “Deng et al. (2023) devised a two-stage method for identifying target elements within intricate HTML. Recently, Gur et al. (2023a) designed a method for interacting with websites via synthesizing programs.”
• Understanding the planning of LLM agents: A survey (2024, methodology): “…include question-answering tasks based on the Wikipedia engine [Yao et al. , 2022] (such as HotPotQA [Yang et al. , 2018] and Fever [Thorne et al. , 2018]) and web browsing tasks to find specific information, including WebShop, Mind2Web [Deng et al. , 2023], and WebArena [Zhou et al. , 2023].”
• Large Language Models for Information Retrieval: A Survey (2023, neutral): “Though the aspect of static search agents has been thoroughly studied, the literature on dynamic search agents remains limited.”
• Cognitive Architectures for Language Agents (2023, background): “, 2022), web manipulation (Yao et al., 2022a; Deng et al., 2023), puzzle solving (Yao et al.”
• WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024, methodology): “More recently, there has been a surge of interest in building more challenging and realistic benchmarks such as Mind2Web (Deng et al., 2023) and We-bArena (Zhou et al., 2023).”