ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts

By Yuta Koreeda, Christopher D. Manning. 2021

186 citations recorded by the source corpus.

Reviewing contracts is a time-consuming procedure that incurs large expenses to companies and social inequality to those who cannot afford it. In this work, we propose"document-level natural language inference (NLI) for contracts", a novel, real-world application of NLI that addresses such problems. In this task, a system is given a set of hypotheses (such as"Some obligations of Agreement may survive termination.") and a contract, and it is asked to classify whether each hypothesis is"entailed by","contradicting to"or"not mentioned by"(neutral to) the contract as well as identifying"evidence"for the decision as spans in the contract. We annotated and release the largest corpus to date consisting of 607 annotated contracts. We then show that existing models fail badly on our task and introduce a strong baseline, which (1) models evidence identification as multi-label classification over spans instead of trying to predict start and end tokens, and (2) employs more sophisticated context segmentation for dealing with long documents. We also show that linguistic characteristics of contracts, such as negations by exceptions, are contributing to the difficulty of this task and that there is much room for improvement.

Subfields: Artificial Intelligence, Political Science and International Relations, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yi Tay
2. Mostafa Dehghani
3. Donald Metzler
4. Denny Zhou
5. Luke S. Zettlemoyer
6. Jonathan Berant
7. Mohit Bansal
8. Percy Liang

In Political Science and International Relations

1. Neil Houlsby

In Natural Language Processing

1. Yi Tay
2. Jonathan Berant
3. Donald Metzler
4. Mohit Bansal
5. Mostafa Dehghani
6. Omer Levy
7. Dara Bahri
8. Denny Zhou

Citations in the corpus

• SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, background): “3 6,737 ContractNLI (Koreeda and Manning, 2021) NLI Legal EM 1,708 1.”
• UL2: Unifying Language Learning Paradigms (2022, methodology): “Table 11 – continued from previous page CosmosQA Accuracy Test Lourie et al. 91.8 91.6(l) Winogrande XL Accuracy Test Lourie et al. 91.3 90.1(l) DocNLI Accuracy Test Qin et al. 76.9 88.2 AdversarialNLI (r3) Accuracy Test Wang et al. 47.7 53.5 VitaminC Accuracy Test Schuster et al. 90.8 91.1 Hellaswa”
• Unifying Language Learning Paradigms (2022, methodology): “We use the Scrolls benchmark (Shaham et al., 2022) which comprises of seven component tasks including GovReport (Huang et al., 2021), SumScr (Chen et al., 2021), QMSUm (Zhong et al., 2021), QASPER (Dasigi et al., 2021), NarrativeQA (Ko ˇ cisk ´ y et al., 2018), QuaLITY (Pang et al., 2021), and Contr”
• The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure (2025, unknown): “…et al., 2025): • Legal retrieval tasks: AILACasedocs and AILAStatutes (Bhattacharya et al., 2020), LegalBenchConsumerContractsQA (Wang et al., 2025; Koreeda and Manning, 2021), LegalBenchCorporateLobbying (Guha et al., 2023; Holzenberger and Van Durme, 2021; Lippi et al., 2019; Ravichander et…”