DocNLI: A Large-scale Dataset for Document-level Natural Language Inference
By Wenpeng Yin, Dragomir R. Radev, Caiming Xiong. 2021
111 citations recorded by the source corpus.
Natural language inference (NLI) is formulated as a unified framework for solving various NLP problems such as relation extraction, question answering, summarization, etc. It has been studied intensively in the past few years thanks to the availability of large-scale labeled datasets. However, most existing studies focus on merely sentence-level inference, which limits the scope of NLI's application in downstream NLP problems. This work presents DocNLI -- a newly-constructed large-scale dataset for document-level NLI. DocNLI is transformed from a broad range of NLP problems and covers multiple genres of text. The premises always stay in the document granularity, whereas the hypotheses vary in length from single sentences to passages with hundreds of words. Additionally, DocNLI has pretty limited artifacts which unfortunately widely exist in some popular sentence-level NLI datasets. Our experiments demonstrate that, even without fine-tuning, a model pretrained on DocNLI shows promising performance on popular sentence-level benchmarks, and generalizes well to out-of-domain NLP tasks that rely on inference at document granularity. Task-specific fine-tuning can bring further improvements. Data, code, and pretrained models can be found at https://github.com/salesforce/DocNLI.
Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yi Tay
- Mohit Bansal
- Luke S. Zettlemoyer
- Mostafa Dehghani
- Jianfeng Gao
- Donald Metzler
- Yejin Choi
- Graham Neubig
In Natural Language Processing
- Yi Tay
- Donald Metzler
- Chenguang Zhu
- Graham Neubig
- Noah Constant
- Peng Liu
- Yang Liu
- Daniel Matthew Cer
In Evaluation & Benchmarks
- Noah Constant
- Benjamin Van Durme
- Chenguang Zhu
- Dara Bahri
- Vered Cohen
- Rami Al-Rfou
- Xavier García
- Heng Ji
Citations in the corpus
- SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, methodology): “• Natural language inference (NLI), including ANLI (Nie et al., 2020), CB, DOCNLI (Yin et al., 2021), MNLI, QNLI, RTE, and SNLI (Bowman et al., 2015).”
- Unifying Language Learning Paradigms (2022, background): “…(Mihaylov et al., 2018), TweetQA (Xiong et al., 2019), QuAIL (Rogers et al., 2020), IMDB (Maas et al., 2011), Agnews (Zhang et al., 2015), DocNLI (Yin et al., 2021), Adversarial NLI (Nie et al., 2019), VitaminC (Schuster et al., 2021a), Civil Comments and Wikipedia Toxicity detection datasets…”
- TRUE: Re-evaluating Factual Consistency Evaluation (2022, background): “As QA and NLI models may struggle with long inputs (Koˇciský et al., 2018; Pang et al., 2021; Yin et al., 2021; Shaham et al., 2022), metrics based on them may fail when handling long text.”
- Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, methodology): “The task of NLI is to determine whether a “hypothesis” is true (entailment), false (contradiction), or undetermined (neutral) under a “premise”.”
- The NLP Task Effectiveness of Long-Range Transformers (2022, methodology): “(Weischedel et al., 2011) and extractive question answering (eQA) (Joshi et al., 2017); 2) sequence classification : natural language inference (NLI) (Yin et al., 2021); 3 and 3) seq2seq : summarization (Summ.)”