LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

By Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael James Bommarito II, Ion Androutsopoulos, Daniel Martin Katz, Nikolaos Aletras. Annual Meeting of the Association for Computational Linguistics, 2021

516 citations recorded by the source corpus.

Laws and their interpretations, legal arguments and agreements are typically expressed in writing, leading to the production of vast corpora of legal text. Their analysis, which is at the center of legal practice, becomes increasingly elaborate as these collections grow in size. Natural language understanding (NLU) technologies can be a valuable tool to support legal practitioners in these endeavors. Their usefulness, however, largely depends on whether current state-of-the-art models can generalize across various tasks in the legal domain. To answer this currently open question, we introduce the Legal General Language Understanding Evaluation (LexGLUE) benchmark, a collection of datasets for evaluating model performance across a diverse set of legal NLU tasks in a standardized way. We also provide an evaluation and analysis of several generic and legal-oriented models demonstrating that the latter consistently offer performance improvements across multiple tasks.

Subfields: Law, Political Science and International Relations

View Paper: https://www.semanticscholar.org/paper/fd33e77884e69f6bc099990fc2790248af2749d9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Law

1. Ilias Chalkidis
2. Dan Jurafsky
3. Peter Henderson

In Political Science and International Relations

1. Christopher D. Manning
2. Ilias Chalkidis
3. Dan Jurafsky

Citations in the corpus

• GPT-4 passes the bar exam (2024, background): “Indeed, part of the charge of legal education is, in fact, a linguistic immersion program where students are trained to parse both the syntactic and semantic nuances of various legal texts.”
• Revisiting Transformer-based Models for Long Document Classification (2022, methodology): “Results in Table 8 show that our results are higher than the ones reported in (Chalkidis et al., 2022 9.8 A comparison between evenly splitting and splitting based on document structure Figure 9 shows that splitting by the paragraph level document structure does not improve performance on the ECtHR…”
• Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset (2022, background): “A number of prior works have pretrained smaller models on smaller subsets of legal data, including private data that is subject to restrictive licenses [30, 129].”
• UK-PRBENCH: A Paragraph-Level Precedent Retrieval Benchmark for United Kingdom Case Law (2026, unknown): “This gap is also evident in widely used English legal NLP benchmarks such as LexGLUE (Chalkidis et al., 2022), which do not include any datasets representing the UK legal system.”
• Building Legal Reward Models for Grounding and Abstention (2026, unknown): “Legal NLP has evolved from classification and judgement prediction toward large-scale reasoning benchmarks such as LexGLUE, LegalBench, LEXam, and PLAWBench (Chalkidis et al., 2022; Guha et al., 2023; Fan et al., 2026; Shi et al., 2026).”
• The Large Language Model GreekLegalRoBERTa (2024, unknown): “…of the Transformer architecture [32] has led to the creation of many large language models (LLMs) and related benchmarks for the legal domain [7–9 , For the Greek language in particular, [1] studied the problem of named entity recognition (NER) in Greek legal documents us-ing various kinds of…”