LEGAL-BERT: “Preparing the Muppets for Court’”

By Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, Ion Androutsopoulos. Findings, 2020

484 citations recorded by the source corpus.

BERT has achieved impressive performance in several NLP tasks. However, there has been limited investigation on its adaptation guidelines in specialised domains. Here we focus on the legal domain, where we explore several approaches for applying BERT models to downstream legal tasks, evaluating on multiple datasets. Our findings indicate that the previous guidelines for pre-training and fine-tuning, often blindly followed, do not always generalize well in the legal domain. Thus we propose a systematic investigation of the available strategies when applying BERT in specialised domains. These are: (a) use the original BERT out of the box, (b) adapt BERT by additional pre-training on domain-specific corpora, and (c) pre-train BERT from scratch on domain-specific corpora. We also propose a broader hyper-parameter search space when fine-tuning for downstream tasks and we release LEGAL-BERT, a family of BERT models intended to assist legal NLP research, computational law, and legal technology applications.

Subfields: Artificial Intelligence, Political Science and International Relations

View Paper: https://www.semanticscholar.org/paper/f6245b3e6270e4dc2e279c4b728030523dffcff4

Citations in the corpus

• Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents (2021, unknown): “Besides, some works conduct continued pre-training on legal documents to bridge the gap between the generic domain and the legal domain (Zhong et al., 2019; Chalkidis et al., 2020).”
• Assessing the performance gap between lexical and semantic models for information retrieval with formulaic legal language (2025, unknown): “We represent the documents with the top-K frequent N-grams of the training set, where 𝐾 = 5 , 000 and 𝑁 ∈ [ 1 , 2 ] .”