Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts

By Hanmeng Liu, Leyang Cui, Jian Liu, Yue Zhang. 2020

59 citations recorded by the source corpus.

Natural language inference (NLI) is a fundamental NLP task, investigating the entailment relationship between two texts.
 Popular NLI datasets present the task at sentence-level. While adequate for testing semantic representations, they fall short for testing contextual reasoning over long texts, which is a natural part of the human inference process. We introduce ConTRoL, a new dataset for ConTextual Reasoning over Long texts. Consisting of 8,325 expert-designed "context-hypothesis" pairs with gold labels, ConTRoL is a passage-level NLI dataset with a focus on complex contextual reasoning types such as logical reasoning. It is derived from competitive selection and recruitment test (verbal reasoning test) for police recruitment, with expert level quality. Compared with previous NLI benchmarks, the materials in ConTRoL are much more challenging, involving a range of reasoning types. Empirical results show that state-of-the-art language models perform by far worse than educated humans. Our dataset can also serve as a testing-set for downstream tasks like checking the factual correctness of summaries.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Nan Duan
2. Ming Zhou
3. Daya Guo
4. Duyu Tang
5. Swarat Chaudhuri
6. Kaj Bostrom
7. Jianfeng Gao
8. Daxin Jiang

In Computer Vision and Pattern Recognition

1. Nan Duan
2. Ming Zhou
3. Swarat Chaudhuri
4. Kaj Bostrom
5. Jian Yin
6. Fabian M. Suchanek
7. Chadi Helwe
8. Greg Durrett

In Natural Language Processing

1. Nan Duan
2. Ming Zhou
3. Daya Guo
4. Swarat Chaudhuri
5. Kaj Bostrom
6. Greg Durrett
7. Duyu Tang
8. Jian Yin

Citations in the corpus

• Flexible Generation of Natural Language Deductions (2021, background): “Recent datasets have been explicitly designed to test deduction ability (Liu et al., 2020; Yu et al., 2020; Holzenberger et al., 2020) and new types of models take inspiration from formal and informal reasoning (Clark et al.”
• Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text (2021, background): “Liu et al. (2021) propose ConTRoL dataset investigating NLI for long texts and examining more complex contextual reasoning types.”
• Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4 (2023, background): “ConTRoL (Liu et al., 2020) is an NLI dataset that further investigates contextual reasoning under the NLI framework.”
• Analytical Reasoning of Text (2022, background): “Logical Reasoning The task of Natural Language Inference (NLI) (Dagan et al., 2005; Bowman et al., 2015; Wang et al., 2019; Williams et al., 2018; Welleck et al., 2019; Khot et al., 2018; Nie et al., 2020; Bhagavatula et al., 2020; Liu et al., 2020a) requires the models to detect the logical entailm”
• Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning (2021, background): “Several other benchmarks in this vein also show bad performance: ReClor [Yu et al., 2020], QuAIL [Rogers et al., 2020a], ConTRoL [Liu et al., 2020a], Strate-6 gyQA [Geva et al., 2021], AR-LSAT [Zhong et al., 2021], and CLUTRR [Sinha et al., 2019].”