When does pretraining help?: assessing self-supervised learning for law and the CaseHOLD dataset of 53,000+ legal holdings

By Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, Daniel E. Ho. International Conference on Artificial Intelligence and Law, 2021

362 citations recorded by the source corpus.

While self-supervised learning has made rapid advances in natural language processing, it remains unclear when researchers should engage in resource-intensive domain-specific pretraining (domain pretraining). The law, puzzlingly, has yielded few documented instances of substantial gains to domain pretraining in spite of the fact that legal language is widely seen to be unique. We hypothesize that these existing results stem from the fact that existing legal NLP tasks are too easy and fail to meet conditions for when domain pretraining can help. To address this, we first present CaseHOLD (Case Holdings On Legal Decisions), a new dataset comprised of over 53,000+ multiple choice questions to identify the relevant holding of a cited case. This dataset presents a fundamental task to lawyers and is both legally meaningful and difficult from an NLP perspective (F1 of 0.4 with a BiLSTM baseline). Second, we assess performance gains on CaseHOLD and existing legal NLP datasets. While a Transformer architecture (BERT) pretrained on a general corpus (Google Books and Wikipedia) improves performance, domain pretraining (on a corpus of ≈3.5M decisions across all courts in the U.S. that is larger than BERT's) with a custom legal vocabulary exhibits the most substantial performance gains with CaseHOLD (gain of 7.2% on F1, representing a 12% improvement on BERT) and consistent performance gains across two other legal tasks. Third, we show that domain pretraining may be warranted when the task exhibits sufficient similarity to the pretraining corpus: the level of performance increase in three legal tasks was directly tied to the domain specificity of the task. Our findings inform when researchers should engage in resource-intensive pretraining and show that Transformer-based architectures, too, learn embeddings suggestive of distinct legal language.

Subfields: Artificial Intelligence, Political Science and International Relations

View Paper: https://www.semanticscholar.org/paper/ef03f3910685dda98279aa2bcefbeada3362a626

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Political Science and International Relations

Citations in the corpus

ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts (2021, methodology): “….357.039 .834.002 DeBERTa v2xlarge None .933.002 .859.008 .885.001 .360.027 .855.002 BERTbase Pretrained from scratch using a case law corpus (Zheng et al., 2021) .870.015 .578.052 .831.032 .289.026 .783.040 BERTbase Fine-tuned on case law and contract corpora (Chalkidis et al., 2020)…”
GPT-4 passes the bar exam (2024, neutral): “20,21 37 The Uniform Bar Exam”
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset (2022, background): “A number of prior works have pretrained smaller models on smaller subsets of legal data, including private data that is subject to restrictive licenses [30, 129].”
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English (2021, background): “…al., 2019, 2020; Zhong et al., 2020b; Bommarito et al., 2021), from judgment prediction (Aletras et al., 2016; Sim et al., 2016; Katz et al., 2017; Zhong et al., 2018; Chalkidis et al., 2019a; Malik et al., 2021), information extraction from legal documents (Chalkidis et al., 2018, 2019c; Chen et…”
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (2023, neutral): “For example, on the OVERRULING dataset, FrugalGPT achieves a 1% accuracy gain while reducing costs by 73% compared to the best LLM API, GPT-4.”
The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure (2025, unknown): “…(Wang et al., 2025; Koreeda and Manning, 2021), LegalBenchCorporateLobbying (Guha et al., 2023; Holzenberger and Van Durme, 2021; Lippi et al., 2019; Ravichander et al., 2019; Wang et al., 2023; Wilson et al., 2016; Zheng et al., 2021; Zimmeck et al., 2019), LegalSummarization (Manor and Li, 2019).”