A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis

By Izzeddin Gur, Hiroki Furuta, Austin L. Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, Aleksandra Faust. International Conference on Learning Representations, 2023

391 citations recorded by the source corpus.

Pre-trained large language models (LLMs) have recently achieved better generalization and sample efficiency in autonomous web automation. However, the performance on real-world websites has still suffered from (1) open domainness, (2) limited context length, and (3) lack of inductive bias on HTML. We introduce WebAgent, an LLM-driven agent that learns from self-experience to complete tasks on real websites following natural language instructions. WebAgent plans ahead by decomposing instructions into canonical sub-instructions, summarizes long HTML documents into task-relevant snippets, and acts on websites via Python programs generated from those. We design WebAgent with Flan-U-PaLM, for grounded code generation, and HTML-T5, new pre-trained LLMs for long HTML documents using local and global attention mechanisms and a mixture of long-span denoising objectives, for planning and summarization. We empirically demonstrate that our modular recipe improves the success on real websites by over 50%, and that HTML-T5 is the best model to solve various HTML understanding tasks; achieving 18.7% higher success rate than the prior method on MiniWoB web automation benchmark, and SoTA performance on Mind2Web, an offline task planning evaluation.

Subfields: Artificial Intelligence, Information Systems, Signal Processing

View Paper: https://www.semanticscholar.org/paper/a53c8ba374d430d6c3786d13c04edb200d547750

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Signal Processing

Citations in the corpus

WebArena: A Realistic Web Environment for Building Autonomous Agents (2023, background): “Gur et al. (2023) propose a web agent that decomposes tasks into more manageable sub-tasks and synthesizes Javascript code for the task executions.”
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024, methodology): “Recent studies have explored the construction of text-based web browsing environments and how to instruct large language model agents to perform web navigation (Nakano et al., 2021; Gur et al., 2023; Zhou et al., 2023; Lu et al., 2023).”
Large Language Models for Information Retrieval: A Survey (2023, neutral): “It is crucial to address these challenges to effectively realize the potential of LLMs in this paradigm.”
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024, background): “Several recent work have also explored the use of strong VLMs for building visual agents for mobile platforms (Zhan & Zhang, 2023; Chu et al., 2023; Yang et al., 2023b) as well as for the web (Gur et al., 2023; Hong et al., 2023).”
Cognitive Architectures for Language Agents (2023, background): “, 2023b), and websites (Shi et al., 2017; Nakano et al., 2021; Yao et al., 2022a; Zhou et al., 2023; Gur et al., 2023; Deng et al., 2023) as well as general code execution (Yang et al.”
Group-in-Group Policy Optimization for LLM Agent Training (2025, neutral): “The use of large language models (LLMs) as autonomous agents has expanded rapidly across domains such as program generation [22], smart device operation [23, 24, 25, 26], interactive gameplay [10], and robot behavior control [27].”
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (2024, positive): “To strike a balance and avoid meaningless excessive resolutions, we enlarge the allowed input sizes to 36 ViT (Dosovitskiy et al., 2021) slices, and use CLIP@224px (Radford et al., 2021) as the image encoder for more flexible splitting, pushing the maximum supported resolution to 1 , 344 × 1 , 344…”
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, positive): “Multimodal Large Language Models (MLLMs) [1, 4, 15, 34, 42, 49, 49, 78], which integrate language and vision, exhibit strong capacities to think and reason in open-ended dialog and practical tasks like web agents [21, 28, 33, 35].”
GPT-4V(ision) is a Generalist Web Agent, if Grounded (2024, methodology): “Many works have focused on improving web agents relying on the HTML document (Deng et al., 2023; Gur et al., 2023; 2022; 2023; Kim et al., 2023; Sridhar et al., 2023).”
AppAgent: Multimodal Agents as Smartphone Users (2023, neutral): “The related work of Wu et al. [39] introduces the Never-ending UI Learner, an autonomous app crawler that learns UI interaction tasks by programmatically tapping UI elements and observing the effects, avoiding the need for costly human labeling.”
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (2024, methodology): “Recent advances in Large Language Models (LLMs) such as GPT-4 (OpenAI, 2023) and LLaMA (Tou-vron et al., 2023) have significantly propelled the evolution of GUI agents (Gur et al., 2023a; Zhou et al., 2023).”
The rise and potential of large language model based agents: a survey (2025, positive): “In task-oriented deployment, the agent follows high-level instructions from users, undertaking tasks such as goal decomposition [167,171,293,294], sequence planning of sub-goals [167,295], interactive exploration of the environment [165,292,296,297], until the ﬁnal objective is achieved.”