Question-Answer Driven Semantic Role Labeling: Using Natural Language to Annotate Natural Language

By Luheng He, Michael Lewis, Luke S. Zettlemoyer. 2015

239 citations recorded by the source corpus.

This paper introduces the task of questionanswer driven semantic role labeling (QA-SRL), where question-answer pairs are used to represent predicate-argument structure. For example, the verb “introduce” in the previous sentence would be labeled with the questions “What is introduced?”, and “What introduces something?”, each paired with the phrase from the sentence that gives the correct answer. Posing the problem this way allows the questions themselves to define the set of possible roles, without the need for predefined frame or thematic role ontologies. It also allows for scalable data collection by annotators with very little training and no linguistic expertise. We gather data in two domains, newswire text and Wikipedia articles, and introduce simple classifierbased models for predicting which questions to ask and what their answers should be. Our results show that non-expert annotators can produce high quality QA-SRL data, and also establish baseline performance levels for future work on this task.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “Finally, as a check to our hypothesis, we also tried a curriculum schedule that used SST, QA-SRL, QA-ZRE, WOZ, WikiSQL and MWSC in the initial curriculum.”
Unifying Question Answering and Text Classification via Span Extraction (2019, methodology): “…(Rajpurkar et al., 2016), QA Zero-shot Relationship Extraction (ZRE; we use the 0 th split) (Levy et al., 2017), QA Semantic Role Labeling (SRL) (He et al., 2015), Commonsense Question Answering (CQA) (Tal-mor et al., 2018) and the two versions (Web and Wiki) of TriviaQA (Joshi et al., 2017).”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “Other work has used QA-based task formulation for more-targeted applications including semantic role labeling (He et al., 2015), relation extraction (Levy et al., 2017), coreference resolution (Wu et al., 2020) and named entity recognition (Li et al., 2020) as question answering.”
Zero-Shot Relation Extraction via Reading Comprehension (2017, background): “For the task of identifying predicate-argument structures, QA-SRL (He et al., 2015) was proposed as an open schema for semantic roles, in which the relation between an argument and a predicate is expressed as a natural-language question containing the predicate (“Where was someone educated?”) whose…”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…et al. 2019 piqa other Bisk et al. 2020 poem_sentiment cls/sentiment analysis Sheng and Uthus 2020 proto_qa other Boratko et al. 2020 qa_srl other He et al. 2015 qasc qa/multiple-choice qa Khot et al. 2020 quail qa/multiple-choice qa Rogers et al. 2020 quarel qa/multiple-choice qa Tafjord et…”
Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task (2017, methodology): “In the interest of balancing these considerations and testing the effectiveness of different tasks, we ran two tasks in parallel: sentiment analysis and question-answer driven semantic role labeling (QA-SRL; He et al., 2015).”
Distill and Replay for Continual Language Learning (2020, background): “We use Stanford Question Answering Dataset (SQuAD, Rajpurkar et al. (2016)) for question answering task, Stanford Sentiment Treebank (SST, Radford et al. (2017)) for sentiment analysis, WikiSQL (Zhong et al. (2017)) for Semantic Parsing, English Wizard of Oz (WOZ) for task-oriented dialogue, and QA-”
Neural Module Networks for Reasoning over Text (2019, background): “Being formalism-free and close to an end-user task, QA is increasingly becoming a proxy for gauging a model’s natural language understanding capability (He et al., 2015; Talmor et al., 2018).”
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020, methodology): “QA-SRL The question-answer driven semantic role labeling dataset (QA-SRL; He et al., 2015) for a QA task that is derived from a semantic role labeling task.”
The Value of Semantic Parse Labeling for Knowledge Base Question Answering (2016, positive): “One promising approach is to use a more dialog-driven interface using natu-ral language (similar to (He et al., 2015)).”
Transforming Question Answering Datasets Into Natural Language Inference Datasets (2018, background): “…likely stem from the fact that question answering and question formulation are organic tasks that people perform in daily life — making QA data easy to crowd-source (He et al., 2015), and easy to ﬁnd in well-designed pre-existing resources such as reading comprehension exams (Lai et al., 2017b).”
Crowdsourcing Question-Answer Meaning Representations (2017, methodology): “In addition to the semantic formalisms (Palmer et al., 2005; Meyers et al., 2004; Banarescu et al., 2013; He et al., 2015) we have already discussed, FrameNet (Baker et al., 1998) also focuses predicate-argument structure, but has more ﬁne-grained argument types.”
CorefQA: Coreference Resolution as Query-based Span Prediction (2019, background): “He et al. (2015) showed that semantic role labeling annotations could be solicited by using question-answer pairs to represent the predicate-argument structure.”
TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions (2020, methodology): “Motivated by recent works (He et al., 2015; Michael et al., 2017; Levy et al., 2017; Gard-ner et al., 2019b), we propose using natural language question answering as an annotation format for temporal relations.”
Controlled Crowdsourcing for High-Quality QA-SRL Annotation (2019, methodology): “Previous attempts to annotate QA-SRL initially involved trained annotators (He et al., 2015) but later resorted to crowdsourcing (Fitzgerald et al.”
Unifying Question Answering, Text Classification, and Regression via Span Extraction (2019, methodology): “A Zero-shot Relationship Extraction (ZRE; we use the 0th split and append the token unanswerable to all examples so it can be extracted as a span) (Levy et al.,2017), QA Semantic Role Labeling (SRL) (He et al., 2015), Commonsense Question Answering (CQA; we use version 1:0) (Talmor et al.,2018) and ”
Supervised Open Information Extraction (2018, background): “Stanovsky and Dagan (2016) created a large Open IE corpus ( OIE2016 ) for verbal predicates by automatic conversion from QA-SRL (He et al., 2015), a variant of traditional SRL that labels arguments of verbs with simple, template-based natural language questions.”