Transforming Question Answering Datasets Into Natural Language Inference Datasets

By Dorottya Demszky, Kelvin Guu, Percy Liang. 2018

187 citations recorded by the source corpus.

Existing datasets for natural language inference (NLI) have propelled research on language understanding. We propose a new method for automatically deriving NLI datasets from the growing abundance of large-scale question answering datasets. Our approach hinges on learning a sentence transformation model which converts question-answer pairs into their declarative forms. Despite being primarily trained on a single QA dataset, we show that it can be successfully applied to a variety of other QA resources. Using this system, we automatically derive a new freely available dataset of over 500k NLI examples (QA-NLI), and show that it exhibits a wide range of inference phenomena rarely seen in previous NLI datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, background): “Converting short-answer or multiple choice questions into entailment examples, as we do when experimenting with transfer learning, has been proposed in several prior works (Demszky et al., 2018; Poliak et al., 2018; Khot et al., 2018).”
KERMIT: Generative Insertion-Based Modeling for Sequences (2019, methodology): “For this experiment, we use the human-annotated QA2D dataset assembled by Demszky et al. (2018), which consists of examples from the SQuAD dataset (Rajpurkar et al., 2016) in which the answer has been extended from a single phrase into a full declarative sentence.”
DART: Open-Domain Structured Data Record to Text Generation (2020, methodology): “High quality natural language questions in open domain semantic parsing datasets such as WikiSQL and QA2D techniques found in automatically constructing NLI datasets (Demszky et al., 2018) present themselves as an attractive opportunity to semi-automatically construct an abundance of declarative sen”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, methodology): “To train the question generator, we adapt the QA2D dataset Demszky et al. (2018).”
Are Red Roses Red? Evaluating Consistency of Question-Answering Models (2019, methodology): “Instead, we use the QA2D system of Demszky et al. (2018) to transform a ( q, a ) into declarative form d , and then use the dependency parse of d to extract questions about the subject ( Subj ), direct object ( Dobj ), adjectival modiﬁers ( Amod ), or prepositional phrases ( Prep ) (Table 1).”
Explaining Answers with Entailment Trees (2021, methodology): “, using the QA2D model (Demszky et al., 2018) the only valid entailment tree constructable from that input.”
Memory-Based Model Editing at Scale (2022, methodology): “For the QA-hard setting, we generate harder in-scope examples that test logically entailed facts (ze = WHO IS THE UK PM? BORIS JOHNSON → xin = WHERE IS BORIS JOHNSON THE PM?) or true/false questions (xin = TRUE OR FALSE: THERESA MAY IS THE UK PM) using automated techniques (Demszky et al., 2018; Rib”
Looking Beyond Sentence-Level Natural Language Inference for Question Answering and Text Summarization (2021, methodology): “…with fixed or learned aggregations over segmented premises to perform a target downstream task with long contexts (Falke et al., 2019; Trivedi et al., 2019); and (2) work addressing the need for task-specific NLI datasets (Kryściński et al., 2019; Demszky et al., 2018; Welleck et al., 2019).”
DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference (2021, methodology): “Second, we consider the QANLI dataset (Demszky et al., 2018) which recasts question answering into NLI.”
WebArena: A Realistic Web Environment for Building Autonomous Agents (2023, background): “5-turbo-0613 to assess whether â could entail each of the key concepts in a∗ [6].”
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021, background): “The GLUE (Wang et al., 2018) benchmark is a collection of 9 datasets for evaluating natural language understanding systems, of which we use Corpus of Linguistic Acceptability (CoLA) (Warstadt et al., 2019b), Stanford Sentiment Treebank (SST) (Socher et al., 2013), Microsoft Research Paragraph Corpus”
Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answering (2020, methodology): “Declarative form For a chain to support the answer to a question, we construct H as a declarative form of the question + answer using standard QA2D tools, e.g., (Demszky et al., 2018).”
Exploiting Reasoning Chains for Multi-hop Science Question Answering (2021, methodology): “We use the rulebased model of Demszky et al. (2018) to generate the hypothesis.”
Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models (2020, background): “Thus we expand the usual textual entailment-based deﬁnition to also include MRC, as answering a question can be framed as ﬁnding an answer that is entailed by the question and the provided context, and the tasks can be transformed vice versa (Demszky et al., 2018).”
Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options (2020, methodology): “Similar to ours, both DNC and QA-NLI use automatic methods to generate sentence pairs.”
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, methodology): “Concretely, questions and answers are combined into declarative claims using a sequence-to-sequence model (Song, 2022; Demszky et al., 2018).”
Case-Based Abductive Natural Language Inference (2020, methodology): “To perform the experiments, we transform each question-candidate answer pair into a hypothesis following the methodology described in (Demszky et al., 2018).”