Mapping natural language commands to web elements
By Panupong Pasupat, Tianrui Jiang, Evan Zheran Liu, Kelvin Guu, Percy Liang. 2018
52 citations recorded by the source corpus.
The web provides a rich, open-domain environment with textual, structural, and spatial properties. We propose a new task for grounding language in this environment: given a natural language command (e.g., “click on the second article”), choose the correct element on the web page (e.g., a hyperlink or text box). We collected a dataset of over 50,000 commands that capture various phenomena such as functional references (e.g. “find who made this site”), relational reasoning (e.g. “article by john”), and visual reasoning (e.g. “top-most article”). We also implemented and analyzed three baseline models that capture different phenomena present in the dataset.
Subfields: Artificial Intelligence, Natural Language Processing, LLM Agents & Tool Use
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Bing Liu
- Sahisnu Mazumder
- Sergey Levine
- Mohit Bansal
- Kate Saenko
- Trevor Darrell
- Jianfeng Gao
- Shuai Wang
In Natural Language Processing
- Bing Liu
- Shuai Wang
- Stephen Marcus McAleer
- Pierre Baldi
- Geunwoo Kim
- Sahisnu Mazumder
- Sepideh Esmaeilpour
- Andrea Burns
In LLM Agents & Tool Use
- Stephen Marcus McAleer
- Pierre Baldi
- Aleksandra Faust
- Mustafa Safdari
- Geunwoo Kim
- Izzeddin Gur
- Bing Liu
Citations in the corpus
- Language Models can Solve Computer Tasks (2023, background): “1 Automated computer tasks The automation of computer tasks is an important topic for both information retrieval and natural language processing [32, 39, 38, 24, 50].”
- Interactive Mobile App Navigation with Uncertain or Under-specified Natural Language Commands (2022, background): “Prior work in automating web tasks [24, 28] limit user interaction to a single screen, unlike MoTIF which contains task demos with a median of 8 visited screens (# views in Table 1).”
- Understanding HTML with Large Language Models (2022, background): “Alongside these benchmarks, many works have developed models for web navigation and related subtasks (Pasupat et al., 2018; Bommasani et al., 2021; He et al., 2021; Gur et al., 2021; Humphreys et al., 2022; Liu et al., 2018; Jia et al., 2019).”
- An Application-Independent Approach to Building Task-Oriented Chatbots with Interactive Continual Learning (2020, methodology): “[34, 29] designed an NLI for submitting web forms and interacting with webpages and [22] proposed an NLI for Bash commands.”
- FLIN: A Flexible Natural Language Interface for Web Navigation (2020, unknown): “Then, a scoring function (Pasupat et al., 2018) or a neural policy (Liu et al., 2018a) are trained to identify which UI element best supports a given command.”