CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review

By Dan Hendrycks, Collin Burns, Anya Chen, Spencer Ball. NeurIPS Datasets and Benchmarks, 2021

341 citations recorded by the source corpus.

Many specialized domains remain untouched by deep learning, as large labeled datasets require expensive expert annotators. We address this bottleneck within the legal domain by introducing the Contract Understanding Atticus Dataset (CUAD), a new dataset for legal contract review. CUAD was created with dozens of legal experts from The Atticus Project and consists of over 13,000 annotations. The task is to highlight salient portions of a contract that are important for a human to review. We find that Transformer models have nascent performance, but that this performance is strongly influenced by model design and training dataset size. Despite these promising results, there is still substantial room for improvement. As one of the only large, specialized NLP benchmarks annotated by experts, CUAD can serve as a challenging research benchmark for the broader NLP community.

Subfields: Artificial Intelligence, Political Science and International Relations

View Paper: https://www.semanticscholar.org/paper/6a1b25f7a67395ad1e676027322913acbb0a0635

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Political Science and International Relations

Citations in the corpus

DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, neutral): “, 2018) 11M (1M) CONTRACTS† 500 commercial legal contracts (Hendrycks et al., 2021) 1.”
Jamba: A Hybrid Transformer-Mamba Language Model (2024, methodology): “Specifically, we evaluated the models on the following datasets: NarrativeQA (QA on narratives; [25]), LongFQA (finance; [2]), Natural Questions (NQ; Wikipedia; [26]), CUAD (law; [21]), and SFiction (science fiction).”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, neutral): “, 2021) 1M (1M) BREAKING NEWS† 20K English news articles, scraped using (Ou-Yang, Lucas) 11M (1M) CONTRACTS† 500 commercial legal contracts (Hendrycks et al., 2021) 1.”
A Survey on Evaluation of Large Language Models (2023, background): “, 2023) Comprehensive domain knowledge General language task Overall performance across multiple benchmarks CUAD (Hendrycks et al., 2021b) Legal contract review Specific downstream task Legal contract understanding TRUSTGPT (Huang et al.”
Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents (2021, neutral): “Thus, LegalAI has received great attention from both natural language processing (NLP) researchers and legal professionals (Zhong et al., 2018, 2020b; Wu et al., 2020; Chalkidis et al., 2020; Hendrycks et al., 2021).”
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset (2022, background): “[55] Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball.”
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English (2021, methodology): “…2018; Chalkidis et al., 2019a; Malik et al., 2021), information extraction from legal documents (Chalkidis et al., 2018, 2019c; Chen et al., 2020; Hendrycks et al., 2021) and case summarization (Bhattacharya et al., 2019) to legal question answering (Ravichander et al., 2019; Kien et al., 2020;…”
ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts (2021, methodology): “Using a model pretrained on a case law corpus (Zheng et al., 2021) CUAD (Hendrycks et al., 2021) gave marginal improvement on NLI, making it the best performing model on the ContractNLI dataset.”