A data-driven approach for learning to control computers

By Peter Conway Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro and 1 more. 2022

128 citations recorded by the source corpus.

It would be useful for machines to use computers as humans do so that they can aid us in everyday tasks. This is a setting in which there is also the potential to leverage large-scale expert demonstrations and human judgements of interactive behaviour, which are two ingredients that have driven much recent success in AI. Here we investigate the setting of computer control using keyboard and mouse, with goals specified via natural language. Instead of focusing on hand-designed curricula and specialized action spaces, we focus on developing a scalable method centered on reinforcement learning combined with behavioural priors informed by actual human-computer interactions. We achieve state-of-the-art and human-level mean performance across all tasks within the MiniWob++ benchmark, a challenging suite of computer control problems, and find strong evidence of cross-task transfer. These results demonstrate the usefulness of a unified human-agent interface when training machines to use computers. Altogether our results suggest a formula for achieving competency beyond MiniWob++ and towards controlling computers, in general, as a human would.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “While there are other works on web navigation and computer-control, most of them assume the typical human interface, that takes as input images of a computer screen and output keyboard commands in order to solve digital tasks (Shi et al., 2017; Gur et al., 2019; 2021; Toyama et al., 2021; Humphreys ”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “One approach to grounding language models in interaction is by learning downstream networks with pre-trained LLM representations [38, 22, 21, 39, 40, 41, 42, 43].”
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, methodology): “Several follow-up papers have tackled MiniWoB using techniques like workflow-guided exploration [29], curriculum and meta-learning [15], DOM tree representation [21], adversarial environment generation [16] and large-scale behavioral cloning [20].”
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (2024, unknown): “…understanding and generation capabilities of (multimodal) LLMs empower GUI agents to generalize beyond simple simulated settings (Shi et al., 2017; Humphreys et al., 2022) to diverse and complex real-world environments, including the web (Deng et al., 2023; Zhou et al., 2024; Yao et al., 2022),…”
Understanding HTML with Large Language Models (2022, methodology): “…HTML understanding is to train specialized models, i.e., gathering application-speciﬁc datasets and designing neural network (NN) architectures to leverage inductive biases of the HTML’s structure; see, e.g., Liu et al. (2018); Toyama et al. (2021); Gur et al. (2019, 2021); Humphreys et al. (2022).”
Language Models can Solve Computer Tasks (2023, methodology): “We generously estimate the performance of CC-Net (RL) based on their figures.”
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (2024, methodology): “Beyond testing agent robustness, the dynamic construction of tasks supports the use of online learning methodologies, particularly reinforcement learning [45, 29, 19, 14].”
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, methodology): “We use 12K demonstrations [42], and compare HTML-T5 among supervised-finetuned baselines [24, 28].”
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, background): “20,21 In this paper, we seek to extend the paradigm of training large, general-purpose foundation models to sequential decision domains by utilizing freely available internet-scale unlabeled video datasets with a simple semi-supervised imitation learning method.”
Android in the Wild: A Large-Scale Dataset for Android Device Control (2023, methodology): “On the web front, previous studies have developed RL [20, 25, 29, 18], behavioral cloning [24], and LLM-based models [19, 26, 17].”
Teaching Large Language Models to Self-Debug (2023, background): “, 2023) demonstrates a related method combined with state and agent grounding that allows a large language model to accomplish computer control tasks in the MiniWoB++ benchmark (Humphreys et al., 2022), and improves zero-shot performance on several math and commonsense reasoning benchmarks.”