LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

By Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher R'e, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin M. K. Peters, Brandon Waldon, D. Rockmore and 30 more. Social Science Research Network, 2023

460 citations recorded by the source corpus.

The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LegalBench: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LegalBench was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning -- which distinguish between its many forms -- correspond to LegalBench tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LegalBench, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LegalBench enables.

Subfields: Law, Political Science and International Relations

View Paper: https://www.semanticscholar.org/paper/0aa5f3e94fab50e3c19870880fdaf245bfebdfae

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Jiawei Han
2. Heng Ji
3. Dan Roth
4. Mark Yatskar
5. Mohit Bansal
6. Bowen Jin
7. Meng Jiang
8. Chaitanya Malaviya

Citations in the corpus

• Large Language Models on Graphs: A Comprehensive Survey (2023, background): “The representations learned on such a graph with both text and structure information can be utilized for clause classification [100] and opinion recommendation [101].”
• Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (2024, unknown): “Unlike the LegalBench version of this task, which is multiple-choice for easier evaluation, this is presented as an open-ended task.”
• ExpertQA: Expert-Curated Questions and Attributed Answers (2023, background): “, 2022), legal (Guha et al., 2023), technology (Dos Santos et al.”
• Query-efficient model evaluation using cached responses (2026, unknown): “Specifically, we consider four tasks: MATH (Hendrycks et al., 2021), which includes 7 subject-based subtasks; LegalBench (Guha et al., 2023), which includes 11 legal reasoning subtasks; MedQA (Jin et al., 2020), a multiple-choice medical exam dataset; and WMT-14 (Bojar et al., 2014), which includes…”