Crowdsourcing Question-Answer Meaning Representations

By Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, Luke S. Zettlemoyer. 2017

81 citations recorded by the source corpus.

We introduce Question-Answer Meaning Representations (QAMRs), which represent the predicate-argument structure of a sentence as a set of question-answer pairs. We develop a crowdsourcing scheme to show that QAMRs can be labeled with very little training, and gather a dataset with over 5,000 sentences and 100,000 questions. A qualitative analysis demonstrates that the crowd-generated question-answer pairs cover the vast majority of predicate-argument relationships in existing datasets (including PropBank, NomBank, and QA-SRL) along with many previously under-resourced ones, including implicit arguments and relations. We also report baseline models for question generation and answering, and summarize a recent approach for using QAMR labels to improve an Open IE system. These results suggest the freely available QAMR data and annotation scheme should support significant future work.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Break It Down: A Question Understanding Benchmark (2020, background): “Recent work has focused on cheaply eliciting quality annotations from nonexperts through crowdsourcing (He et al., 2016; Iyer et al., 2017; Michael et al., 2018).”
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (2019, methodology): “QAMR (Michael et al., 2018) To construct the Question-Answer Meaning Representation (QAMR) dataset, crowdworkers are presented with an English sentence along with target nonstopwords from the sentence.”
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020, background): “QAMR The Question–Answer Meaning Representations dataset (Michael et al., 2018) is a crowdsourced QA task consisting of question–answer pairs that correspond to predicate–argument relationships.”
Transforming Question Answering Datasets Into Natural Language Inference Datasets (2018, background): “The questions vary greatly in terms of their type and difﬁculty: questions in QAMR hinge on selecting the right arguments within a single sentence, while questions in RACE, written for middle-and high-schoolers, require holistic reasoning about the text (e.g.”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, background): “Since this representation is based on natural language, it is both interpretable and highly flexible (Michael et al., 2017; Weiss et al., 2021).”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, unknown): “It was shown in particular that RoBERTA trained on QAMR [119] achieved the best results for argument extraction.”
Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2021, methodology): “Our approach builds on previous work using question-answer pairs as a meaning representation (He et al., 2015; Michael et al., 2018), and more generally from the tradition of using question answering ability as a proxy for language understanding (Lehnert, 1977; Hirschman et al., 1999; Peñas et al.,…”
Asking without Telling: Exploring Latent Ontologies in Contextual Representations (2020, background): “…ontology matching to identify further cases of transferable features, or perhaps the emergence of categories signifying pipelined reasoning (Tenney et al., 2019a), surface patterns, or new, perhaps unexpected distinctions which can appear when going beyond existing schemas (Michael et al., 2018).”
Can Small and Synthetic Benchmarks Drive Modeling Innovation? A Retrospective Study of Question Answering Modeling Approaches (2021, methodology): “Manual examination of generated QAMR passages and questions suggests that this case is rather uncommon, but it may still introduce a small amount of noise into the benchmark.”
Attributed Text Generation via Post-hoc Research and Revision (2022, background): “Since this representation is based on natural language, it is both interpretable and highly ﬂexible (Michael et al., 2017; Weiss et al., 2021).”
TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions (2020, methodology): “Motivated by recent works (He et al., 2015; Michael et al., 2017; Levy et al., 2017; Gard-ner et al., 2019b), we propose using natural language question answering as an annotation format for temporal relations.”
Supervised Open Information Extraction (2018, methodology): “As described by Michael et al. (2018), QAMR annotations were gathered via crowdsourcing in a two-stage pipeline over Wikipedia and Wikinews text.”
Evaluating Factuality in Generation with Dependency-level Entailment (2020, methodology): “It contains 373 test samples, each containing an input source sentence from CNN/DM and two summary sentences covering the same content generated using the model from Chen and Bansal (2018).”
Large-Scale QA-SRL Parsing (2018, methodology): “Some more recent semantic resources require less annotator training, or can be crowdsourced (Abend and Rappoport, 2013; Reisinger et al., 2015; Basile et al., 2012; Michael et al., 2018).”
Comparing Test Sets with Item Response Theory (2021, background): “QAMR The Question-Answer Meaning Repre-sentations (Michael et al., 2018) is a QA dataset where the question-answer pairs are created from sentences’ predicate-argument relationships.”
Domain-agnostic Question-Answering with Adversarial Training (2019, background): “, 2013), QAMR (MR) (Michael et al., 2017), QAST (ST) (Jitkrittum et al.”
Question Answering is a Format; When is it Useful? (2019, background): “QASRL / QAMR The motivation for this work is explicitly to make annotation cheaper by having crowd workers label semantic dependencies in sentences using natural language instead of training them in a formalism ( He et al., 2015; FitzGerald et al., 2018; Michael et al., 2018).”
Partial Or Complete, That’s The Question (2019, unknown): “answering, known to provide only partial coverage are successful examples of this idea (He et al., 2015; Michael et al., 2017).”