FLIN: A Flexible Natural Language Interface for Web Navigation

By Sahisnu Mazumder, Oriana Riva. 2020

29 citations recorded by the source corpus.

AI assistants can now carry out tasks for users by directly interacting with website UIs. Current semantic parsing and slot-filling techniques cannot flexibly adapt to many different websites without being constantly re-trained. We propose FLIN, a natural language interface for web navigation that maps user commands to concept-level actions (rather than low-level UI actions), thus being able to flexibly adapt to different websites and handle their transient nature. We frame this as a ranking problem: given a user command and a webpage, FLIN learns to score the most relevant navigation instruction (involving action and parameter values). To train and evaluate FLIN, we collect a dataset using nine popular websites from three domains. Our results show that FLIN was able to adapt to new websites in a given domain.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yu Su
2. Shunyu Yao
3. Karthik Narasimhan
4. John Yang
5. Howard Chen
6. Huan Sun
7. Aleksandra Faust
8. Hiroki Furuta

In Computer Vision and Pattern Recognition

1. Karthik Narasimhan
2. Shunyu Yao
3. Huan Sun
4. Aleksandra Faust
5. Yu Su
6. Hiroki Furuta
7. Jihyung Kil
8. Howard Chen

In Natural Language Processing

1. Shunyu Yao
2. Karthik Narasimhan
3. John Yang
4. Howard Chen
5. Huan Sun
6. Yu Su
7. Boyuan Zheng
8. Douglas Eck

Citations in the corpus

• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “Some benchmarks only contain either a single classification task [39, 46, 31] or interactions containing only a handful of different pages in each episode [43].”
• Understanding HTML with Large Language Models (2022, unknown): “…simulated websites for training web-based agents, with application to task fulfillment (Yao et al., 2022; Gur et al., 2021; Burns et al., 2022; Mazumder and Riva, 2020; Shi et al., 2017; Liu et al., 2018) as well as information retrieval or question-answering (Adolphs et al., 2021; Nogueira…”
• GPT-4V(ision) is a Generalist Web Agent, if Grounded (2024, background): “We showed that LMMs present a great promise for generalist web agents, with a success rate of 50 % on live websites given an oracle grounding method.”
• A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, background): “…decision making task where agents manipulate browsers following given instructions (Shi et al., 2017), such as form filling (Diaz et al., 2013) or information retrieval (Adolphs et al., 2022) through the sequence of computer actions (Li et al., 2020; Mazumder & Riva, 2020; Shvo et al., 2021).”