DOM-Q-NET: Grounded RL on Structured Language

By Sheng Jia, Jamie Ryan Kiros, Jimmy Ba. 2019

41 citations recorded by the source corpus.

Building agents to interact with the web would allow for significant improvements in knowledge understanding and representation learning. However, web navigation tasks are difficult for current deep reinforcement learning (RL) models due to the large discrete action space and the varying number of actions between the states. In this work, we introduce DOM-Q-NET, a novel architecture for RL-based web navigation to address both of these problems. It parametrizes Q functions with separate networks for different action categories: clicking a DOM element and typing a string input. Our model utilizes a graph neural network to represent the tree-structured HTML of a standard web page. We demonstrate the capabilities of our model on the MiniWoB environment where we can match or outperform existing work without the use of expert demonstrations. Furthermore, we show 2x improvements in sample efficiency when training in the multi-task setting, allowing our model to transfer learned behaviours across tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, methodology): “Several follow-up papers have tackled MiniWoB using techniques like workflow-guided exploration [29], curriculum and meta-learning [15], DOM tree representation [21], adversarial environment generation [16] and large-scale behavioral cloning [20].”
Interactive Mobile App Navigation with Uncertain or Under-specified Natural Language Commands (2022, background): “MoTIF subsumes several datasets and topics: web task automation [12, 17, 20, 24, 28], VLN [3, 6, 11], task relevance prediction [10, 13, 27], app design [7, 8, 21]; see Table 1 for comparison.”
The Klarna Product Page Dataset: A Realistic Benchmark for Web Representation Learning (2021, methodology): “GCN-GRU: This model is inspired by the local embedding module in the DOM-Q-Net algorithm [15].”
Android in the Wild: A Large-Scale Dataset for Android Device Control (2023, methodology): “On the web front, previous studies have developed RL [20, 25, 29, 18], behavioral cloning [24], and LLM-based models [19, 26, 17].”
Understanding HTML with Large Language Models (2022, background): “Alongside these benchmarks, many works have developed models for web navigation and related subtasks (Pasupat et al., 2018; Bommasani et al., 2021; He et al., 2021; Gur et al., 2021; Humphreys et al., 2022; Liu et al., 2018; Jia et al., 2019).”
Language Models can Solve Computer Tasks (2023, background): “We are most excited about gains in productivity in science and education, including AI research, which will lead to even faster development of new beneficial technologies and treatments.”
A data-driven approach for learning to control computers (2022, methodology): “Shown are previously published MiniWob++ results (Gur et al., 2018; Jia et al., 2019; Liu et al., 2018; Shi et al., 2017), which each only report performance for a subset of tasks (for these results the performance on non-reported tasks is set to zero, all scores used are given in Appendix Table 3).”
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, background): “Prior works have realized the web automation via reinforcement learning (Gur et al., 2019; Humphreys et al., 2022; Jia et al., 2019; Shaw et al., 2023), finetuned (Furuta et al.”