Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences

By Hongyuan Mei, Mohit Bansal, Matthew R. Walter. 2015

247 citations recorded by the source corpus.

We propose a neural sequence-to-sequence model for direction following, a task that is essential to realizing effective autonomous agents. Our alignment-based encoder-decoder model with long short-term memory recurrent neural networks (LSTM-RNN) translates natural language instructions to action sequences based upon a representation of the observable world state. We introduce a multi-level aligner that empowers our model to focus on sentence "regions" salient to the current world state by using multiple abstractions of the input sentence. In contrast to existing methods, our model uses no specialized linguistic resources (e.g., parsers) or task-specific annotations (e.g., seed lexicons). It is therefore generalizable, yet still achieves the best results reported to-date on a benchmark single-sentence dataset and competitive results for the limited-training multi-sentence setting. We analyze our model through a series of ablations that elucidate the contributions of the primary components of our model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, LLM Agents & Tool Use

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Sergey Levine
  3. Jianfeng Gao
  4. Yoshua Bengio
  5. Percy Liang
  6. Luke S. Zettlemoyer
  7. Yejin Choi
  8. Quoc V. Le

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Trevor Darrell
  3. Dacheng Tao
  4. Jitendra Malik
  5. Xiaogang Wang
  6. Cordelia Schmid
  7. Andrew Zisserman
  8. Ming-Hsuan Yang

In Natural Language Processing

  1. Mohit Bansal
  2. Yejin Choi
  3. Jianfeng Gao
  4. Luke S. Zettlemoyer
  5. Percy Liang
  6. Richard Socher
  7. Christopher D. Manning
  8. Mirella Lapata

In LLM Agents & Tool Use

  1. Shimon Whiteson
  2. Mohit Bansal
  3. Percy Liang
  4. Mirella Lapata
  5. Li Dong
  6. Robin Jia
  7. Sergey Levine
  8. Jacob Andreas

Citations in the corpus