Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
By Hongyuan Mei, Mohit Bansal, Matthew R. Walter. 2015
247 citations recorded by the source corpus.
We propose a neural sequence-to-sequence model for direction following, a task that is essential to realizing effective autonomous agents. Our alignment-based encoder-decoder model with long short-term memory recurrent neural networks (LSTM-RNN) translates natural language instructions to action sequences based upon a representation of the observable world state. We introduce a multi-level aligner that empowers our model to focus on sentence "regions" salient to the current world state by using multiple abstractions of the input sentence. In contrast to existing methods, our model uses no specialized linguistic resources (e.g., parsers) or task-specific annotations (e.g., seed lexicons). It is therefore generalizable, yet still achieves the best results reported to-date on a benchmark single-sentence dataset and competitive results for the limited-training multi-sentence setting. We analyze our model through a series of ablations that elucidate the contributions of the primary components of our model.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, LLM Agents & Tool Use
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Sergey Levine
- Jianfeng Gao
- Yoshua Bengio
- Percy Liang
- Luke S. Zettlemoyer
- Yejin Choi
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Jitendra Malik
- Xiaogang Wang
- Cordelia Schmid
- Andrew Zisserman
- Ming-Hsuan Yang
In Natural Language Processing
- Mohit Bansal
- Yejin Choi
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Richard Socher
- Christopher D. Manning
- Mirella Lapata
In LLM Agents & Tool Use
- Shimon Whiteson
- Mohit Bansal
- Percy Liang
- Mirella Lapata
- Li Dong
- Robin Jia
- Sergey Levine
- Jacob Andreas
Citations in the corpus
- Learning to Understand Goal Specifications by Modelling Reward (2018, background): “Andreas & Klein (2015); Mei et al. (2016) sidestep the intermediate formal representation and train a Conditional Random Field (CRF) and a sequence-to-sequence neural model respectively to directly predict the actions from the demonstrations.”
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “A large number of prior works have learned language-conditioned behavior via imitation learning [51, 22, 20, 13, 26, 37] or reinforcement learning [52, 53, 49, 54, 55, 56, 21, 41].”
- Visual Dialog (2016, background): “• Robotics applications ( e.g . search and rescue missions) where the operator may be ‘situationally blind’ and operating via language [40] (Human: ‘Is there smoke in any room around you?’”
- Data Recombination for Neural Semantic Parsing (2016, methodology): “Mei et al. (2016) use an RNN model to perform a related task of instruction following.”
- Language to Logical Form with Neural Attention (2016, methodology): “Mei et al. (2016) use a sequence-to-sequence model to map navigational instructions to actions.”
- Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “While a number of approaches [33, 34, 55] have been proposed for the various navigation benchmarks, they generally employ a single model that learns to map directly from instructions to actions from a limited corpus of annotated trajectories.”
- What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment (2015, positive): “3 Our model aligns based on multiple abstractions of the input: both the original input record as well as the hidden annotations mj = (r> j ;h > j ) >, an approach that has previously been shown to yield better results than aligning based only on the hidden state (Mei et al., 2015).”
- Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, background): “There has been a line of work for building agents that can execute natural language instructions: Tellex et al. (2011; 2014) for robotics and MacMa-hon et al. (2006); Chen and Mooney (2011); Mei et al. (2015) for a simulated environment.”
- A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “A more flexible approach is to use a single policy that conditions on the currently executed instruction, allowing some generalization to unseen instructions [Mei et al. , 2016; Oh et al. , 2017].”
- BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning (2018, background): “There has been much research on instruction following with natural language (Tellex et al., 2011; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Mei et al., 2016; Williams et al., 2018) as well as several datasets including SAIL (Macmahon et al., 2006; Chen and Mooney, 2011) and Room-to-Room…”
- Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout (2019, background): “This task has been studied by many works (Tellex et al., 2011; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Andreas and Klein, 2015; Mei et al., 2016; Misra et al., 2017) in recent years.”
- Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation (2018, background): “[27] proposed a sequence-to-sequence model to map the language to navigation actions.”
- Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, background): “Extensive efforts have been made to ground language instructions into a sequence of actions (MacMahon et al., 2006; Branavan et al., 2009; Vogel & Jurafsky, 2010; Tellex et al., 2011; Artzi & Zettlemoyer, 2013; Andreas & Klein, 2015; Mei et al., 2016; Cohn et al., 2016; Misra et al., 2017).”
- Hierarchical Decision Making by Generating and Following Natural Language Instructions (2019, unknown): “More recently, neural models instruction following have been developed for a variety of domains, for example [11] and [10].”
- Shifting the Baseline: Single Modality Performance on Visual Navigation & QA (2018, background): “Modern approaches use neural architectures to map natural language to the (simulated) world and execute actions (Paxton et al., 2019; Chen et al., 2018; Nguyen et al., 2018; Blukis et al., 2018; Fried et al., 2018; Mei et al., 2016).”
- Learning to Follow Language Instructions with Adversarial Reward Induction (2018, background): “[2, 16] sidestep the intermediate formal representation and train a Conditional Random Field (CRF) and a sequence-to-sequence neural model respectively to directly predict the actions from the demonstrations.”
- A Tale of Two DRAGGNs: A Hybrid Approach for Interpreting Action-Oriented and Goal-Oriented Instructions (2017, background): “Recent approaches leveraging deep neural networks have formulated the language grounding problem as sequence-to-sequence learning or multi-label clas-sification (Mei et al., 2016; Arumugam et al., 2017).”