SciTaiL: A Textual Entailment Dataset from Science Question Answering

By Tushar Khot, Ashish Sabharwal, Peter Clark. 2018

525 citations recorded by the source corpus.

We present a new dataset and model for textual entailment, derived from treating multiple-choice question-answering as an entailment problem. SciTail is the first entailment set that is created solely from natural sentences that already exist independently ``in the wild'' rather than sentences authored specifically for the entailment task. Different from existing entailment datasets, we create hypotheses from science questions and the corresponding answer candidates, and premises from relevant web sentences retrieved from a large corpus. These sentences are often linguistically challenging. This, combined with the high lexical similarity of premise and hypothesis for both entailed and non-entailed pairs, makes this new entailment task particularly difficult. The resulting challenge is evidenced by state-of-the-art textual entailment systems achieving mediocre performance on SciTail, especially in comparison to a simple majority class baseline. As a step forward, we demonstrate that one can improve accuracy on SciTail by 5% using a new neural model that exploits linguistic structure.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Adversarial Training for Large Neural Language Models (2020, methodology): “To assess the combination of adversarial pre-training and ﬁne-tuning, we follow Jiang et al. (2019) and use MNLI (Williams et al., 2018) (from GLUE), ANLI, SWAG (Zellers et al., 2018), SNLI (Bowman et al., 2015), SciTail (Khot et al., 2018).”
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, methodology): “Following what was done in SciTail (Khot et al., 2018), we convert questions and answer-options to statements by either substituting the answer-option for the blanks in ﬁll-in-the-blank questions, or appending a separator token and the answer-option to the question.”
Multi-Task Deep Neural Networks for Natural Language Understanding (2019, methodology): “SciTail This is a textual entailment dataset derived from a science question answering (SciQ) dataset (Khot et al., 2018).”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, methodology): “We perform textual entailment experiments on the Science enTailment dataset SciTail (Khot et al., 2018).”
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, methodology): “We adapted two neural entailment models, DecompAttn (Parikh et al., 2016) and DGEM (Khot et al., 2018), to the task of answering multiple-choice questions.”
Improving Question Answering with External Knowledge (2019, methodology): “…and textual information from Wikipedia by first retrieving relevant documents in Wikipedia and then running a MRC model to extract a text span from the documents based on the question (Chen et al., 2017; Wang et al., 2018b; Kratzwald and Feuerriegel, 2018; Lee et al., 2018; Lin et al., 2018).”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, neutral): “, 2018), SciTail (Khot et al., 2018), SICK (Marelli et al.”
Hypothesis Only Baselines in Natural Language Inference (2018, background): “SciTail Recently released, SciTail is an NLI dataset created from 4 th grade science questions and multiple-choice answers (Khot et al., 2018).”
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences (2018, methodology): “The SciTail dataset (Khot et al., 2018), created from science exams, is somewhat different from the two datasets, being smaller (27,026 examples), and labeled only as entailment or neutral.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…Lee 2005 samsum cg/summarization Gliwa et al. 2019 scicite cls/other Cohan et al. 2019 sciq qa/multiple-choice qa Welbl et al. 2017 scitail cls/nli Khot et al. 2018 search_qa qa/closed-book qa Dunn et al. 2017 sick cls/nli Marelli et al. 2014 sms_spam cls/other Almeida et al. 2011 social_i_qa…”
Show Your Work: Improved Reporting of Experimental Results (2019, background): “3) show that the different models require different budgets to reach their reported performance in expectation, ranging from 2 (n-gram) to 20 (DGEM).”
An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, methodology): “(13) SciTail (Khot et al., 2018); (14) Implicit Discourse Relation Classiﬁcation (Prasad et al., 2019; Kim et al., 2020); and (15) Emotion Detection (Saravia et al., 2018).”
Dialogue Natural Language Inference (2018, background): “The resulting sentence pairs are thus drawn from a natural dialogue domain that differs from existing NLI datasets, which are either drawn from different domains such as image captions or use synthetic templating [2, 7, 11, 16, 19, 20].”
FLEX: Unifying Evaluation for Few-Shot NLP (2021, background): “In contrast, domain transfer keeps the same classes between meta-training and meta-testing but changes the textual domain (e.g., generalizing from MNLI to science-focused SciTail [4, 21]).”