A Test Collection for Evaluating Legal Case Law Search

By Daniel Locke, Guido Zuccon. 2018

63 citations recorded by the source corpus.

Test collection based evaluation represents the standard of evalua- tion for information retrieval systems. Legal IR, more speci cally case law retrieval, has no such standard test collection for evalua- tion. In this paper, we present a test collection for use in evaluating case law search, being the retrieval of judicial decisions relevant to a particular legal question. The collection is made available at ielab.io/caselaw.

Subfields: Artificial Intelligence, Information Systems, Political Science and International Relations, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jimmy J. Lin
2. Zhiyuan Liu
3. Andrew Yates
4. Maosong Sun
5. Rodrigo Nogueira
6. Ion Androutsopoulos
7. Chaojun Xiao
8. Jamie Callan

In Information Systems

1. Rodrigo Nogueira
2. Andrew Yates
3. Jimmy J. Lin
4. Zhiyuan Liu
5. Maosong Sun
6. Ion Androutsopoulos
7. Prodromos Malakasiotis
8. Cunchao Tu

In Political Science and International Relations

1. Maosong Sun
2. Cunchao Tu
3. Zhiyuan Liu
4. Chaojun Xiao
5. Ilias Chalkidis
6. Ion Androutsopoulos
7. Tianyang Zhang
8. Nikolaos Aletras

In Information Retrieval & Search

1. Jimmy J. Lin
2. Andrew Yates
3. Rodrigo Nogueira
4. Zhiyuan Liu
5. Zeynep Akkalyoncu Yilmaz
6. Sebastian Hofstätter
7. Maosong Sun
8. Zhuyun Dai

In Evaluation & Benchmarks

1. Zhiyuan Liu
2. Rodrigo Nogueira
3. Andrew Yates
4. Maosong Sun
5. Jimmy J. Lin
6. Ion Androutsopoulos
7. Prodromos Malakasiotis
8. Nikolaos Aletras

Citations in the corpus

• Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “s (e.g., law review articles). While certainly not as plentiful as newswire test collections, there are resources available for searching legal documents. A recently built test collection is CaseLaw [Locke and Zuccon, 2018], which comprises approximately 4M judicial decisions in the United States (s”
• How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence (2020, background): “There are several LegalIR datasets, including COLIEE (Kano et al., 2018), CaseLaw (Locke and Zuccon, 2018), and CM (Xiao et al., 2019).”
• LexGLUE: A Benchmark Dataset for Legal Language Understanding in English (2021, background): “We also initially considered some very interesting legal Information Retrieval (IR) datasets (Locke and Zuccon, 2018; Chalkidis et al., 2021b) that aim to examine crucial real-life tasks (relevant case law retrieval, regulatory compliance).”
• Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases (2021, background): “Paragraph-level rationale extraction can be also conceived as selfsupervised extractive summarization to denoise long documents, a direction we plan to explore in the challenging task of case law retrieval (Locke and Zuccon, 2018).”