Learning to Interpret Natural Language Navigation Instructions from Observations

By David L. Chen, Raymond J. Mooney. 2011

545 citations recorded by the source corpus.

The ability to understand natural-language instructions is critical to building intelligent agents that interact with humans. We present a system that learns to transform natural-language navigation instructions into executable formal plans. Given no prior linguistic knowledge, the system learns by simply observing how humans follow navigation instructions. The system is evaluated in three complex virtual indoor environments with numerous objects and landmarks. A previously collected realistic corpus of complex English navigation instructions for these environments is used for training and testing data. By using a learned lexicon to refine inferred plans and a supervised learner to induce a semantic parser, the system is able to automatically learnto correctly interpret a reasonable fraction of the complex instructions in this corpus.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Learning to Understand Goal Specifications by Modelling Reward (2018, background): “Chen & Mooney (2011); Artzi & Zettlemoyer (2013) train semantic parsers to produce a formal representation of the query that when fed to a predeﬁned execution model matches exactly the sequence of actions from the demonstration.”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “There, research has focused on mapping instructions to actions [5, 13, 31, 35, 48], but these works do not involve visual, interactive environments.”
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks (2015, background): “A: west, north This is related to the work of (Chen & Mooney, 2011).”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “…and Parikh, 2013; Antol et al., 2015), caption generation (Chen et al., 2016), referring expression resolution (Mitchell et al., 2010; Matuszek et al., 2012; FitzGerald et al., 2013), and instruction following (MacMahon et al., 2006; Chen and Mooney, 2011; Bisk et al., 2016; Misra et al., 2018).”
Learning to Automatically Solve Algebra Word Problems (2014, background): “Techniques that consider multiple sentences typically do so in a serial fashion, processing each in turn with limited cross-sentence reasoning (Branavan et al., 2009; Zettlemoyer and Collins, 2009; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013).”
Learning Dependency-Based Compositional Semantics (2011, background): “, 2009; Goldwasser and Roth, 2011), and navigate a map by following directions (Vogel and Jurafsky, 2010; Chen and Mooney, 2011).”
A Joint Model of Language and Perception for Grounded Attribute Learning (2012, background): “Most closely related to our work are approaches that learn probabilistic language models from natural language input (Matuszek et al., 2012; Chen & Mooney, 2011), especially those that include a visual component (Tellex et al., 2011).”
Scaling Semantic Parsers with On-the-Fly Ontology Matching (2013, background): “, 2012; Krishnamurthy and Kollar, 2013), interpreting instructions (Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013), and generating programs (Kushman and Barzilay, 2013).”
Zero-shot Text Classification With Generative Language Models (2019, background): “…translation in low resource languages (Gu et al., 2018), iteratively guiding policies with language (Co-Reyes et al., 2018) for instruction following (Branavan et al., 2009; Chen and Mooney, 2011), and generating WikiSQL-style structured queries from natural language queries (Huang et al., 2018).”
Jointly Learning to Parse and Perceive: Connecting Natural Language to the Physical World (2013, background): “…language acquisition leverages semantic parsing to map from natural language to a formal representation of an environment (Kate and Mooney, 2007; Chen and Mooney, 2008; Shimizu and Haas, 2009; Matuszek relation. et al., 2010; Dzifcak et al., 2009; Cantrell et al., 2010; Chen and Mooney, 2011).”
Modular Multitask Reinforcement Learning with Policy Sketches (2016, background): “Existing work on instruction following focuses on mapping from natural language strings to symbolic action sequences that are then executed by a hard-coded interpreter (Branavan et al., 2009; Chen & Mooney, 2011; Artzi & Zettlemoyer, 2013; Tellex et al., 2011).”
Language to Logical Form with Neural Attention (2016, background): “…relying on logical-from annotations, e.g., from sentences paired with conversational logs (Artzi and Zettlemoyer, 2011), system demonstrations (Chen and Mooney, 2011; Goldwasser and Roth, 2011; Artzi and Zettle-moyer, 2013), question-answer pairs (Clarke et al., 2010; Liang et al., 2013), and…”
Learning Language Games through Interaction (2016, background): “, 2011, 2014), and following instructions (Vogel and Jurafsky, 2010; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013) Semantic parsing utterances to logical forms, which we leverage, plays an important role in these set-”
Gated-Attention Architectures for Task-Oriented Language Grounding (2017, background): “Chen and Mooney (2011) learn a semantic parser to learn an optimal navigation policy in a 2-”
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, background): “There has been a line of work for building agents that can execute natural language instructions: Tellex et al. (2011; 2014) for robotics and MacMa-hon et al. (2006); Chen and Mooney (2011); Mei et al. (2015) for a simulated environment.”
A Roadmap Towards Machine Intelligence (2015, background): “We have several points of contact with the semantic parsing literature, such as navigation tasks in an artificial world (MacMahon et al., 2006) and reward-based learning from natural language instructions (Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013).”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “…use object-level representation and relational modeling to exploit the structure of the instruction in relation to world entities, parsing the language instruction into a formal language [ Kuhlmann et al. , 2004; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Andreas and Klein, 2015 ] .”