Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration

By Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, Percy Liang. 2018

349 citations recorded by the source corpus.

Reinforcement learning (RL) agents improve through trial-and-error, but when reward is sparse and the agent cannot discover successful action sequences, learning stagnates. This has been a notable problem in training deep RL agents to perform web-based tasks, such as booking flights or replying to emails, where a single mistake can ruin the entire sequence of actions. A common remedy is to "warm-start" the agent by pre-training it to mimic expert demonstrations, but this is prone to overfitting. Instead, we propose to constrain exploration using demonstrations. From each demonstration, we induce high-level "workflows" which constrain the allowable actions at each time step to be similar to those in the demonstration (e.g., "Step 1: click on a textbox; Step 2: enter some text"). Our exploration policy then learns to identify successful workflows and samples actions that satisfy these workflows. Workflows prune out bad exploration directions and accelerate the agent's ability to discover rewards. We use our approach to train a novel neural policy designed to handle the semi-structured nature of websites, and evaluate on a suite of web tasks, including the recent World of Bits benchmark. We achieve new state-of-the-art results, and show that workflow-guided exploration improves sample efficiency over behavioral cloning by more than 100x.

Subfields: Artificial Intelligence, Computer Science Applications, LLM Agents & Tool Use, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Graham Neubig
  3. Percy Liang
  4. Jianfeng Gao
  5. Sergey Levine
  6. Yoshua Bengio
  7. Luke S. Zettlemoyer
  8. Quoc V. Le

In Computer Science Applications

  1. Aleksandra Faust
  2. Izzeddin Gur
  3. Jimmy Ba
  4. Doina Precup
  5. Philippe Hamel
  6. Pierre Baldi
  7. Percy Liang
  8. Graham Neubig

In LLM Agents & Tool Use

  1. Aleksandra Faust
  2. Izzeddin Gur
  3. Stephen Marcus McAleer
  4. Pierre Baldi
  5. Geunwoo Kim
  6. Timothy P. Lillicrap
  7. Oriana Riva
  8. Panupong Pasupat

In Reinforcement Learning

  1. Sergey Levine
  2. Ofir Nachum
  3. Pieter Abbeel
  4. Shixiang Shane Gu
  5. Jimmy Ba
  6. Honglak Lee
  7. Chelsea Finn
  8. Timothy P. Lillicrap

Citations in the corpus