Weakly Supervised Semantic Parsing with Abstract Examples

By Omer Goldman, Veronica Latcinnik, Udi Naveh, Amir Globerson, Jonathan Berant. 2017

70 citations recorded by the source corpus.

Training semantic parsers from weak supervision (denotations) rather than strong supervision (programs) complicates training in two ways. First, a large search space of potential programs needs to be explored at training time to find a correct program. Second, spurious programs that accidentally lead to a correct denotation add noise to training. In this work we propose that in closed worlds with clear semantic types, one can substantially alleviate these problems by utilizing an abstract representation, where tokens in both the language utterance and program are lifted to an abstract form. We show that these abstractions can be defined with a handful of lexical rules and that they result in sharing between different examples that alleviates the difficulties in training. To test our approach, we develop the first semantic parser for CNLVR, a challenging visual reasoning dataset, where the search space is large and overcoming spuriousness is critical, because denotations are either TRUE or FALSE, and thus random programs are likely to lead to a correct denotation. Our method substantially improves performance, and reaches 82.5% accuracy, a 14.7% absolute accuracy improvement compared to the best reported accuracy so far.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

STaR: Bootstrapping Reasoning With Reasoning (2022, background): “The idea of alternating between filtering steps and training steps more broadly is also well-grounded in prior work in NLP such as [23] and in other weak supervision contexts [24, 25].”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “Consistency measures the proportion of unique sentences for which predictions are correct for all paired images (Goldman et al., 2018).”
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding (2018, background): “…Andreas et al. [2016] attempted to use the latent structure in language to help question answering and reasoning, Rothe et al. [2017] studied the use of formal programs in modeling human questions, and Goldman et al. [2017] used abstract examples to build weakly-supervised semantic parsers.”
Evaluating NLP Models via Contrast Sets (2020, methodology): “We call this metric contrast consistency, following prior work with related evaluation metrics (Goldman et al., 2018; Suhr and Artzi, 2019).”
TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments (2018, methodology): “We use this to measure consistency over unique sentences, which is measured similar to accuracy, but with a unique sentence considered correct only if all its examples are correct [11].”
QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships (2018, methodology): “This parser uses a type-constrained, encoder-decoder architecture, representative of the current state-of-the-art on many datasets (Krish-namurthy, Dasigi, and Gardner 2017; Yin and Neubig 2017; Goldman et al. 2018).”
Learning an Executable Neural Semantic Parser (2017, background): “2010; Chen and Mooney 2011; Artzi and Zettlemoyer 2013), from abstract examples (Goldman et al. 2018), and from human feedbacks (Iyer et al.”
NLVR2 Visual Bias Analysis (2019, methodology): “We only report accuracy on this subset of the data because consistency (Goldman et al., 2018) changes due to the example selection process and we want to avoid confusion with the consistency measures of the existing splits.”
TAPEX: Table Pre-training via Learning a Neural SQL Executor (2021, background): “, SQL), they have difficulties in training due to the large search space and the presence of spurious programs (Goldman et al., 2018).”
Benchmarking Meaning Representations in Neural Semantic Parsing (2020, background): “…applied to weakly supervised semantic parsing, which receives wide attention because parsers are only supervised with execution results and annotated logical forms are no longer required (Berant et al., 2013; Pasupat and Liang, 2015; Goldman et al., 2018; Liang et al., 2018; Mueller et al., 2019).”
Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs (2019, background): “Similar ideas of using abstract meaning representations have been explored with fully-supervised semantic parsers (Dong and Lapata, 2018; Catherine Finegan-Dollak and Radev, 2018) and in other related tasks (Goldman et al., 2018; Herzig and Berant, 2018; Nye et al., 2019).”
SmBoP: Semi-autoregressive Bottom-up Semantic Parsing (2020, background): “Relational algebra Guo et al. (2019) have shown recently that the mismatch between natu-ral language and SQL leads to parsing difﬁculties.”
Solving math word problems with process- and outcome-based feedback (2022, unknown): “This may depend on both the relative diﬃculty for the model to compute the correct answer with and without reasoning traces, and the lack of spurious solutions (Goldman et al., 2017) in math problems, where incorrect reasoning steps are unlikely to lead to the correct ﬁnal answer.”
LEVER: Learning to Verify Language-to-Code Generation with Execution (2023, background): “As annotating the programs requires domain expertise, for some datasets where the final results can be directly obtained, only z∗ but no y∗ is provided for learning (Artzi & Zettlemoyer, 2013; Cheng & Lapata, 2018; Goldman et al., 2018).”
Semantic Parsing with Dual Learning (2019, background): “Lack of data A semantic parser can be trained from labeled logical forms or weakly supervised samples (Krishnamurthy and Mitchell, 2012; Berant et al., 2013; Liang et al., 2017; Goldman et al., 2018).”