Learning to Navigate the Web

By Izzeddin Gur, Ulrich Rückert, Aleksandra Faust, Dilek Z. Hakkani-Tür. 2018

78 citations recorded by the source corpus.

Learning in environments with large state and action spaces, and sparse rewards, can hinder a Reinforcement Learning (RL) agent's learning through trial-and-error. For instance, following natural language instructions on the Web (such as booking a flight ticket) leads to RL settings where input vocabulary and number of actionable elements on a page can grow very large. Even though recent approaches improve the success rate on relatively simple environments with the help of human demonstrations to guide the exploration, they still fail in environments where the set of possible instructions can reach millions. We approach the aforementioned problems from a different perspective and propose guided RL approaches that can generate unbounded amount of experience for an agent to learn from. Instead of learning from a complicated instruction with a large vocabulary, we decompose it into multiple sub-instructions and schedule a curriculum in which an agent is tasked with a gradually increasing subset of these relatively easier sub-instructions. In addition, when the expert demonstrations are not available, we propose a novel meta-learning framework that generates new instruction following tasks and trains the agent more effectively. We train DQN, deep reinforcement learning agent, with Q-value function approximated with a novel QWeb neural network architecture on these smaller, synthetic instructions. We evaluate the ability of our agent to generalize to new instructions on World of Bits benchmark, on forms with up to 100 elements, supporting 14 million possible instructions. The QWeb agent outperforms the baseline without using any human demonstration achieving 100% success rate on several difficult environments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, LLM Agents & Tool Use

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In LLM Agents & Tool Use

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “While there are other works on web navigation and computer-control, most of them assume the typical human interface, that takes as input images of a computer screen and output keyboard commands in order to solve digital tasks (Shi et al., 2017; Gur et al., 2019; 2021; Toyama et al., 2021; Humphreys ”
WebGPT: Browser-assisted question-answering with human feedback (2021, methodology): “They train their system in two alternative ways: behavior cloning (BC) on synthetically-generated sequences and RL. Finally, there is another body of work that uses BC and RL to control web browsers, for automating other tasks besides question-answering [Shi et al., 2017, Gur et al., 2018].”
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, methodology): “Several follow-up papers have tackled MiniWoB using techniques like workflow-guided exploration [29], curriculum and meta-learning [15], DOM tree representation [21], adversarial environment generation [16] and large-scale behavioral cloning [20].”
Language Models can Solve Computer Tasks (2023, methodology): “The group of models proposed before the publication of [23] are collectively referred to as Others, and include [46, 27, 19, 25].”
Interactive Mobile App Navigation with Uncertain or Under-specified Natural Language Commands (2022, background): “MoTIF subsumes several datasets and topics: web task automation [12, 17, 20, 24, 28], VLN [3, 6, 11], task relevance prediction [10, 13, 27], app design [7, 8, 21]; see Table 1 for comparison.”
Adversarial Environment Generation for Learning to Navigate the Web (2021, methodology): “Following previous work (Shi et al., 2017; Gur et al., 2019; Liu et al., 2018), we formulate web navigation as a sequential decision making problem where we train an agent, parameterized by a network π ( a t | s t ; Θ i ) , that maps an input state s t to output actions a t to maximize the…”
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, background): “Prior works have realized the web automation via reinforcement learning (Gur et al., 2019; Humphreys et al., 2022; Jia et al., 2019; Shaw et al., 2023), finetuned (Furuta et al., 2023; Gur et al., 2022) or prompted LLMs (Kim et al., 2023; Sun et al., 2023; Yao et al., 2022b; Zheng et al., 2023) on…”
Android in the Wild: A Large-Scale Dataset for Android Device Control (2023, methodology): “On the web front, previous studies have developed RL [20, 25, 29, 18], behavioral cloning [24], and LLM-based models [19, 26, 17].”
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (2024, background): “Early research investigated training agents by reinforcement learning (Liu et al., 2018; Gur et al., 2018) on the classic web environment Mini-Wob (Shi et al., 2017).”
The Klarna Product Page Dataset: A Realistic Benchmark for Web Representation Learning (2021, background): “An additional few have investigated the related topic of reinforcement learning over the web [10, 23, 15, 27].”
A data-driven approach for learning to control computers (2022, background): “Shown are previously published MiniWob++ results (Gur et al., 2018; Jia et al., 2019; Liu et al., 2018; Shi et al., 2017), which each only report performance for a subset of tasks (for these results the performance on non-reported tasks is set to zero, all scores used are given in Appendix Table 3).”
Understanding HTML with Large Language Models (2022, methodology): “…HTML understanding is to train specialized models, i.e., gathering application-speciﬁc datasets and designing neural network (NN) architectures to leverage inductive biases of the HTML’s structure; see, e.g., Liu et al. (2018); Toyama et al. (2021); Gur et al. (2019, 2021); Humphreys et al. (2022).”
Rapid Task-Solving in Novel Environments (2020, background): “, 2018) and web navigation (Gur et al., 2018), as well as abstract planning problems of interest in AI research, such as Gregor et al.”