Alignment-Based Compositional Semantics for Instruction Following

By Jacob Andreas, Daniel Klein. 2015

102 citations recorded by the source corpus.

This paper describes an alignment-based model for interpreting natural language instructions in context. We approach instruction following as a search over plans, scoring sequences of actions conditioned on structured observations of text and the environment. By explicitly modeling both the low-level compositional structure of individual actions and the high-level structure of full plans, we are able to learn both grounded representations of sentence meaning and pragmatic constraints on interpretation. To demonstrate the model’s flexibility, we apply it to a diverse set of benchmark tasks. On every task, we outperform strong task-specific baselines, and achieve several new state-of-the-art results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Language Understanding for Text-based Games using Deep Reinforcement Learning (2015, background): “, 2010), navigating with directions (Vogel and Jurafsky, 2010; Kollar et al., 2010; Artzi and Zettlemoyer, 2013; Matuszek et al., 2013; Andreas and Klein, 2015) and playing computer games (Eisenstein et al.”
Learning to Understand Goal Specifications by Modelling Reward (2018, background): “Andreas & Klein (2015); Mei et al. (2016) sidestep the intermediate formal representation and train a Conditional Random Field (CRF) and a sequence-to-sequence neural model respectively to directly predict the actions from the demonstrations.”
Lifelong and Continual Learning Dialogue Systems: Learning during Conversation (2021, background): “The other category (2) focuses on learning a semantic parser to parse the NL command from the user into an intermediate logical form and then, translate the logical form into an executable action in the application (Artzi and Zettlemoyer 2013; Andreas and Klein 2015).”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “Systems that learn to carry out natural language instructions in an interactive environment include approaches based on intermediate structured and executable representations of language [50, 9, 4, 28, 20] and approaches that map directly from language and world state observations to actions [7, 2, ”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “…use object-level representation and relational modeling to exploit the structure of the instruction in relation to world entities, parsing the language instruction into a formal language [ Kuhlmann et al. , 2004; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Andreas and Klein, 2015 ] .”
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning (2017, background): “, 2015, 2016), alignment models (Andreas and Klein, 2015), reinforcement learning (Branavan et al.”
Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences (2015, methodology): “ooney (2011) 54:40 16:18 Chen (2012) 57:28 19:18 Kim and Mooney (2012) 57:22 20:17 Kim and Mooney (2013) 62:81 26:57 Artzi and Zettlemoyer (2013) 65:28 31:93 Artzi, Das, and Petrov (2014) 64:36 35:44 Andreas and Klein (2015) 59:60 – Our model (vDev) 69:98 26:07 Our model (vTest) 71:05 30:34 Primary ”
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout (2019, background): “This task has been studied by many works (Tellex et al., 2011; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Andreas and Klein, 2015; Mei et al., 2016; Misra et al., 2017) in recent years.”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, background): “Extensive efforts have been made to ground language instructions into a sequence of actions (MacMahon et al., 2006; Branavan et al., 2009; Vogel & Jurafsky, 2010; Tellex et al., 2011; Artzi & Zettlemoyer, 2013; Andreas & Klein, 2015; Mei et al., 2016; Cohn et al., 2016; Misra et al., 2017).”
Learning to Follow Language Instructions with Adversarial Reward Induction (2018, background): “[2, 16] sidestep the intermediate formal representation and train a Conditional Random Field (CRF) and a sequence-to-sequence neural model respectively to directly predict the actions from the demonstrations.”
Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms (2018, unknown): “Andreas and Klein (2015b) used a conditional random ﬁeld model to learn alignment between instructions and actions.”
Semantic Parsing with Semi-Supervised Sequential Autoencoders (2016, unknown): “22 Andreas and Klein (2015) 59.”
Robots That Use Language (2020, background): “Others have abstracted the problem to learn the grounding from natural language to subgoals or goals (102, 103).”
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, background): “AI alignment is a ﬁeld that aims to ensure that AI systems adhere to their intended goals, interests, and values, as envisioned by their designers [4, 39, 110, 32, 38, 74, 10].”
Grounding Language for Transfer in Deep Reinforcement Learning (2017, unknown): “…following (Vogel & Jurafsky, 2010; Kollar, Tellex, Roy, & Roy, 2010; Artzi & Zettlemoyer, 2013; Matuszek, Herbst, Zettlemoyer, & Fox, 2013; Andreas & Klein, 2015) and learning to play computer games (Branavan, Silver, & Barzilay, 2011, 2012; Narasimhan, Kulkarni, & Barzilay, 2015; He et…”
Transforming Dependency Structures to Logical Forms for Semantic Parsing (2016, background): “…going through an intermediate semantic representation, e.g., answer-sentence selection (Punyakanok et al., 2004; Heilman and Smith, 2010; Yao et al., 2013) and semantic parsing (Ge and Mooney, 2009; Poon, 2013; Parikh et al., 2015; Xu et al., 2015; Wang et al., 2015; Andreas and Klein, 2015).”
CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society (2023, background): “AI alignment is a ﬁeld that aims to ensure that AI systems adhere to their intended goals, interests, and values, as envisioned by their designers [2, 25, 63, 20, 24, 43, 7].”
Unnatural Language Processing: Bridging the Gap Between Synthetic and Natural Language Data (2020, background): “For grounded language learning in particular, “end-to-end” architectures make it hard to disentangle challenging language learning problems from challenging inference problems of the kind discussed above (Andreas & Klein, 2015).”
Representation Learning for Grounded Spatial Reasoning (2017, background): “Instruction following Spatial reasoning is a common element in many papers on instruction following (MacMahon et al., 2006; Vogel and Jurafsky, 2010; Chen and Mooney, 2011; Artzi and Zettle-moyer, 2013; Kim and Mooney, 2013; Andreas and Klein, 2015).”