Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions

By Yoav Artzi, Luke S. Zettlemoyer. 2013

485 citations recorded by the source corpus.

The context in which language is used provides a strong signal for learning to recover its meaning. In this paper, we show it can be used within a grounded CCG semantic parsing approach that learns a joint model of meaning and context for interpreting and executing natural language instructions, using various types of weak supervision. The joint nature provides crucial benefits by allowing situated cues, such as the set of visible objects, to directly influence learning. It also enables algorithms that learn while executing instructions, for example by trying to replicate human actions. Experiments on a benchmark navigational dataset demonstrate strong performance under differing forms of supervision, including correctly executing 60% more instruction sets relative to the previous state of the art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Semantic Parsing on Freebase from Question-Answer Pairs (2013, background): “, 2011), following navigational instructions (Chen, 2012; Artzi and Zettlemoyer, 2013), and interacting in the real world via perception (Matuszek et al.”
Latent Retrieval for Weakly Supervised Open Domain Question Answering (2019, background): “Our approach is also reminiscent of weakly supervised semantic parsing (Clarke et al., 2010; Liang et al., 2013; Artzi and Zettlemoyer, 2013; Fader et al., 2014; Berant et al., 2013; Kwiatkowski et al., 2013), with which we share similar challenges—(1) inference and learning are tightly coupled, (2)”
A Discrete Hard EM Approach for Weakly Supervised Question Answering (2019, background): “Latent-variable learning has been extensively studied in the literature of semantic parsing (Zettlemoyer and Collins, 2005; Clarke et al., 2010; Liang et al., 2013; Berant et al., 2013; Artzi and Zettlemoyer, 2013).”
Language Understanding for Text-based Games using Deep Reinforcement Learning (2015, background): “, 2010), navigating with directions (Vogel and Jurafsky, 2010; Kollar et al., 2010; Artzi and Zettlemoyer, 2013; Matuszek et al., 2013; Andreas and Klein, 2015) and playing computer games (Eisenstein et al.”
Learning to Understand Goal Specifications by Modelling Reward (2018, background): “Chen & Mooney (2011); Artzi & Zettlemoyer (2013) train semantic parsers to produce a formal representation of the query that when fed to a predeﬁned execution model matches exactly the sequence of actions from the demonstration.”
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback (2018, neutral): “Improving semantic parsers using weak feedback has previously been studied (Goldwasser and Roth (2013); Artzi and Zettlemoyer (2013); inter alia ).”
Grammar-Based Grounded Lexicon Learning (2022, methodology): “There have also been approaches for learning grammatical structures from grounded texts [35, 50, 22, 7, 33, 43].”
Parsing Algebraic Word Problems into Equations (2015, background): “More closely related is work on language grounding, whose goal is the interpretation of a sentence in the context of a world representation (Branavan et al., 2009; Liang et al., 2009; Chen et al., 2010; Bordes et al., 2010; Feng and Lapata, 2010; Hajishirzi et al., 2011; Matuszek et al., 2012; Hajis”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “A series of them parse language instructions into formal logic or rely mainly on lexical analysis to resolve various linguistic ambiguities for embodied agents [2, 33, 34, 51].”
LILA: Language-Informed Latent Actions (2021, background): “A variety of methods have sought to combine language and robotics, spanning approaches that map language to planning primitives [9, 30, 31, 32, 33], perform imitation learning from demonstrations and instructions [11, 34, 35, 36], and pair language instructions with reward functions for…”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “There, research has focused on mapping instructions to actions [5, 13, 31, 35, 48], but these works do not involve visual, interactive environments.”
Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning (2017, background): “…parsing focus on learning parsers without relying on annotated logical forms by leveraging conversational logs (Artzi & Zettlemoyer, 2011), demonstrations (Artzi & Zettlemoyer, 2013), distant supervision (Cai & Yates, 2013; Reddy et al., 2014), and question-answer pairs (Liang et al., 2011).”
Learning to Automatically Solve Algebra Word Problems (2014, background): “Techniques that consider multiple sentences typically do so in a serial fashion, processing each in turn with limited cross-sentence reasoning (Branavan et al., 2009; Zettlemoyer and Collins, 2009; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013).”
The Web as a Knowledge-Base for Answering Complex Questions (2018, background): “Conversely, work on QA through semantic parsing has focused primarily on compositionality: questions are translated to compositional programs that encode a sequence of actions for finding the answer in a knowledge-base (KB) (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Artzi and Zettlemoye”
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task (2018, neutral): “They have been studied extensively (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Wong and Mooney, 2007; Das et al., 2010; Liang et al., 2011; Banarescu et al., 2013; Artzi and Zettlemoyer, 2013; Reddy et al., 2014; Berant and Liang, 2014; Dong and Lapata, 2016).”
Scaling Semantic Parsers with On-the-Fly Ontology Matching (2013, methodology): “…representations (Zettlemoyer and Collins, 2005, 2007; Kwiatkowski et al., 2010, 2011), building derivations to transform the output of the CCG parser based on context (Zettlemoyer and Collins, 2009), and using weakly supervised margin-sensitive parameter updates (Artzi and Zettlemoyer, 2011, 2013).”
Lifelong and Continual Learning Dialogue Systems: Learning during Conversation (2021, background): “The other category (2) focuses on learning a semantic parser to parse the NL command from the user into an intermediate logical form and then, translate the logical form into an executable action in the application (Artzi and Zettlemoyer 2013; Andreas and Klein 2015).”
Modular Multitask Reinforcement Learning with Policy Sketches (2016, background): “Existing work on instruction following focuses on mapping from natural language strings to symbolic action sequences that are then executed by a hard-coded interpreter (Branavan et al., 2009; Chen & Mooney, 2011; Artzi & Zettlemoyer, 2013; Tellex et al., 2011).”