Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
By Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, Percy Liang. 2018
349 citations recorded by the source corpus.
Reinforcement learning (RL) agents improve through trial-and-error, but when reward is sparse and the agent cannot discover successful action sequences, learning stagnates. This has been a notable problem in training deep RL agents to perform web-based tasks, such as booking flights or replying to emails, where a single mistake can ruin the entire sequence of actions. A common remedy is to "warm-start" the agent by pre-training it to mimic expert demonstrations, but this is prone to overfitting. Instead, we propose to constrain exploration using demonstrations. From each demonstration, we induce high-level "workflows" which constrain the allowable actions at each time step to be similar to those in the demonstration (e.g., "Step 1: click on a textbox; Step 2: enter some text"). Our exploration policy then learns to identify successful workflows and samples actions that satisfy these workflows. Workflows prune out bad exploration directions and accelerate the agent's ability to discover rewards. We use our approach to train a novel neural policy designed to handle the semi-structured nature of websites, and evaluate on a suite of web tasks, including the recent World of Bits benchmark. We achieve new state-of-the-art results, and show that workflow-guided exploration improves sample efficiency over behavioral cloning by more than 100x.
Subfields: Artificial Intelligence, Computer Science Applications, LLM Agents & Tool Use, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Graham Neubig
- Percy Liang
- Jianfeng Gao
- Sergey Levine
- Yoshua Bengio
- Luke S. Zettlemoyer
- Quoc V. Le
In Computer Science Applications
- Aleksandra Faust
- Izzeddin Gur
- Jimmy Ba
- Doina Precup
- Philippe Hamel
- Pierre Baldi
- Percy Liang
- Graham Neubig
In LLM Agents & Tool Use
- Aleksandra Faust
- Izzeddin Gur
- Stephen Marcus McAleer
- Pierre Baldi
- Geunwoo Kim
- Timothy P. Lillicrap
- Oriana Riva
- Panupong Pasupat
In Reinforcement Learning
- Sergey Levine
- Ofir Nachum
- Pieter Abbeel
- Shixiang Shane Gu
- Jimmy Ba
- Honglak Lee
- Chelsea Finn
- Timothy P. Lillicrap
Citations in the corpus
- WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, methodology): “These include: 1) generation of good search queries [22, 59] and reformulation [37, 51], 2) strategic exploration for navigating through the website [55, 56, 29], 3) robust language understanding for textual state and action spaces [3, 7, 17, 44], and 4) long-term”
- Mind2Web: Towards a Generalist Agent for the Web (2023, background): “Yet, previous research has been limited by the types of tasks and websites it can handle, either confined to simplified simulation environments [22, 31, 40], or limited to a narrow set of domains and websites [39, 40].”
- Tool Learning with Foundation Models (2023, positive): “Models can be trained to mimic the behavior of human experts through imitation learning (Hussein et al., 2017; Liu et al., 2018b; Baker et al., 2022).”
- AndroidEnv: A Reinforcement Learning Platform for Android (2021, methodology): “This approach was pioneered in the original Atari suite (Bellemare et al., 2013) and has been followed up by a variety of platforms, such as DeepMind Lab (Beattie et al., 2016), OpenAI Universe (OpenAI, 2016) and World of Bits (Liu et al., 2018).”
- GPT-4V(ision) is a Generalist Web Agent, if Grounded (2024, background): “We showed that LMMs present a great promise for generalist web agents, with a success rate of 50 % on live websites given an oracle grounding method.”
- Understanding HTML with Large Language Models (2022, methodology): “…HTML understanding is to train specialized models, i.e., gathering application-specific datasets and designing neural network (NN) architectures to leverage inductive biases of the HTML’s structure; see, e.g., Liu et al. (2018); Toyama et al. (2021); Gur et al. (2019, 2021); Humphreys et al. (2022).”
- Adversarial Environment Generation for Learning to Navigate the Web (2021, methodology): “Following previous work (Shi et al., 2017; Gur et al., 2019; Liu et al., 2018), we formulate web navigation as a sequential decision making problem where we train an agent, parameterized by a network π ( a t | s t ; Θ i ) , that maps an input state s t to output actions a t to maximize the…”
- VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024, methodology): “Raw web page HTML as a Document Object Model (DOM) tree, commonly used in previous work on autonomous web agents (Shi et al., 2017; Liu et al., 2018; Deng et al., 2023).”
- AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework (2023, background): “In practice, many applications demand the presence of agents capable of interacting with environments and making decisions in an online context, such as game playing [34, 46], web interactions [30, 45] and robot manipulations [44].”
- Android in the Wild: A Large-Scale Dataset for Android Device Control (2023, methodology): “0 VH, screen x X x MiniWoB++ [32, 27] synthetic web 17,971 100 2.”
- AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation (2023, neutral): “…many applications require the presence of agents capable of interacting with environments and making decisions in an online context, such as in game playing (Mnih et al., 2013; Vinyals et al., 2017), web interactions (Liu et al., 2018; Shi et al., 2017), and robot manipulations (Shen et al., 2021).”
- FLIN: A Flexible Natural Language Interface for Web Navigation (2020, neutral): “A direct performance comparison with other approaches is not possible as they differ in the type of output (Pasupat et al., 2018) or problem settings (Liu et al., 2018b).”
- SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (2024, methodology): “Early research investigated training agents by reinforcement learning (Liu et al., 2018; Gur et al., 2018) on the classic web environment Mini-Wob (Shi et al., 2017).”
- A data-driven approach for learning to control computers (2022, methodology): “Shown are previously published MiniWob++ results (Gur et al., 2018; Jia et al., 2019; Liu et al., 2018; Shi et al., 2017), which each only report performance for a subset of tasks (for these results the performance on non-reported tasks is set to zero, all scores used are given in Appendix Table 3).”
- Language Models can Solve Computer Tasks (2023, methodology): “MiniWoB++ [36], an extension of MiniWoB [61], is a web-based simulation environment that offers a diverse range of computer tasks, from simple button-clicking to complex compositional tasks requiring advanced reasoning, such as solving math problems.”
- A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, methodology): “Unlike previous works [23, 28, 29, 42], real-world web agents could not pre-define a categorical action space to specify which elements on the websites they should interact.”
- WebArena: A Realistic Web Environment for Building Autonomous Agents (2023, methodology): “Some works rely on static states, limiting agents’ explorations and functional correctness evaluation (Shi et al., 2017; Deng et al., 2023), while others simplify real-world complexities, restricting task variety (Yao et al., 2022a; Liu et al., 2018).”
- Interactive Mobile App Navigation with Uncertain or Under-specified Natural Language Commands (2022, background): “MoTIF subsumes several datasets and topics: web task automation [12, 17, 20, 24, 28], VLN [3, 6, 11], task relevance prediction [10, 13, 27], app design [7, 8, 21]; see Table 1 for comparison.”
- DOM-Q-NET: Grounded RL on Structured Language (2019, methodology): “Results: Figure 3 shows that DOM-Q-NET reaches 100% success rate for most of the tasks selected by Liu et al. (2018), except for click-widget , social-media , and email-inbox .”
- The Klarna Product Page Dataset: A Realistic Benchmark for Web Representation Learning (2021, background): “An additional few have investigated the related topic of reinforcement learning over the web [10, 23, 15, 27].”