BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding

By Emad Elwany, David A. Moore, Gaurav Oberoi. 2019

38 citations recorded by the source corpus.

Fine-tuning language models, such as BERT, on domain specific corpora has proven to be valuable in domains like scientific papers and biomedical text. In this paper, we show that fine-tuning BERT on legal documents similarly provides valuable improvements on NLP tasks in the legal domain. Demonstrating this outcome is significant for analyzing commercial agreements, because obtaining large legal corpora is challenging due to their confidential nature. As such, we show that having access to large legal corpora is a competitive advantage for commercial applications, and academic research on analyzing contracts.

Subfields: Economics and Econometrics, Law, Political Science and International Relations, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Law

In Political Science and International Relations

In Natural Language Processing

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “l., 2016]. There are relatively few transform-based approaches to legal information retrieval, and BERT-based models have only begun to be adopted in legal NLP applications more broadly [Yeung, 2019, Elwany et al., 2019]. Recognizing the importance of community-wide evaluations in driving progress, ”
When does pretraining help?: assessing self-supervised learning for law and the CaseHOLD dataset of 53,000+ legal holdings (2021, methodology): “This is explained in part due to the expense of labeling decisions and challenges around compiling large sets of legal documents [32], leading approaches above to rely on non-English datasets [49, 50] or proprietary datasets [14].”
CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters (2020, background): “While current research mostly focuses on improving language representations for the default “generaldomain”, there seems to be a growing interest in building suitable word embeddings for more specialized domains (El Boukkouri et al., 2019; Si et al., 2019; Elwany et al., 2019).”
Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents (2021, unknown): “Some researchers attempt to transfer the contex-tualized language models pre-trained on the generic domain, such as Wikipedia, Children’s Books, to tasks in the legal domain (Shaghaghian et al., 2020; Zhong et al., 2020b; Shao et al., 2020; Elwany et al., 2019).”