Looking Beyond Sentence-Level Natural Language Inference for Question Answering and Text Summarization

By Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, Kartik Talamadupula. 2021

62 citations recorded by the source corpus.

Natural Language Inference (NLI) has garnered significant attention in recent years; however, the promise of applying NLI breakthroughs to other downstream NLP tasks has remained unfulfilled. In this work, we use the multiple-choice reading comprehension (MCRC) and checking factual correctness of textual summarization (CFCS) tasks to investigate potential reasons for this. Our findings show that: (1) the relatively shorter length of premises in traditional NLI datasets is the primary challenge prohibiting usage in downstream applications (which do better with longer contexts); (2) this challenge can be addressed by automatically converting resource-rich reading comprehension datasets into longer-premise NLI datasets; and (3) models trained on the converted, longer-premise datasets outperform those trained using short-premise traditional NLI datasets on downstream tasks primarily due to the difference in premise lengths.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pascale Fung
2. Greg Durrett
3. Andrea Madotto
4. Wenliang Dai
5. Nayeon Lee
6. Dan Su
7. Zhiting Hu
8. Yan Xu

In Natural Language Processing

1. Greg Durrett
2. Pascale Fung
3. Nayeon Lee
4. Andrea Madotto
5. Etsuko Ishii
6. Yejin Bang
7. Björn Ross
8. Simone Conia

Citations in the corpus

• Survey of Hallucination in Natural Language Generation (2022, background): “[131] find that the low performance of NLI-based metrics is mainly caused by the length of the premises in NLI datasets being shorter than the source documents in abstractive summarization.”
• Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation (2021, methodology): “It is a subset of CNN/Daily Mail dataset (Hermann et al., 2015; Nallapati et al., 2016) and released by Wang et al. (2020).”
• Can NLI Models Verify QA Systems' Predictions? (2021, methodology): “QA-enhanced NLI Model As there might exist different reasoning patterns in the QA datasets which are not covered by the MNLI model (Mishra et al., 2021), we study whether NLI pairs generated from QA datasets can be used jointly with the MNLI data to improve the performance of an NLI model.”
• Biases in Large Language Models: Origins, Inventory, and Discussion (2023, background): “The ield has witnessed a drastic change in paradigm with the advent and wide availability of large-scale pretrained language models, such as BERT [46], GPT [21, 108], T5 [109] and BART [79], which are now pervasive in every high-performance system for Machine Translation [25], Question Answering [95”
• AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, methodology): “…performance when evaluating generative tasks with long texts, NLI-based metrics adopt training sets with long premises (Honovich et al., 2022a; Mishra et al., 2021), use large synthetic datasets (Kryscinski et al., 2020; Utama et al., 2022), or use sentence level evaluation (Laban et al.,…”
• Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024, background): “For example, factual-centred metrics [19, 20, 40, 57] and benchmarks [34, 35, 65] have been proposed to measure and reduce hallucination on specific datasets.”
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “However, Mishra et al. (2021) highlighted that the mismatch in input granularity between conventional NLI datasets and inconsistency detection datasets limits their applicability for effectively detecting inconsistencies.”