COLIEE-2016: Evaluation of the Competition on Legal Information Extraction and Entailment

By Yoshinobu Kano, Mi-Young Kim, Masaharu Yoshioka, Yao Lu, Juliano Rabelo, Naoki Kiyota, Randy Goebel, Ken Satoh. 2016

Subfields: Evaluation & Benchmarks, Information Extraction, Social Sciences & Humanities

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

1. Zhiyuan Liu
2. Mi-Young Kim
3. Randy Goebel
4. Juliano Rabelo
5. Maosong Sun
6. Ion Androutsopoulos
7. Nikolaos Aletras
8. Dan Hendrycks

In Information Extraction

1. Ilias Chalkidis
2. Ion Androutsopoulos
3. Maosong Sun
4. Nikolaos Aletras
5. Zhiyuan Liu

Citations in the corpus

• CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review (2021, methodology): “Kano et al. (2018) evaluate models on multiple tasks for statute law and case law, including information retrieval and entailment/question answering.”
• Overview of COLIEE 2017 (2017, background): “…held the first, second and third competitions on legal information extraction/entailment, COLIEE 2014 , COLIEE 2015 (Kim et al., 2015), COLIEE 2016 (Kim et al., 2016), on a legal data collection, and this helped establish a major experimental effort in the legal information extraction/retrieval…”
• LexGLUE: A Benchmark Dataset for Legal Language Understanding in English (2021, background): “A notable venue for legal QA has been the Competition on Legal Information Extraction and Entailment (COLIEE) (Kim et al., 2016; Kano et al., 2017, 2018).”
• A Summary of the COLIEE 2019 Competition (2019, methodology): “The organizers provided a data set used for previous bar law exams, translated to English [10, 9, 8, 25] as training data (717 questions), with new questions selected from the 2018 bar exam as test data (98 questions).”
• How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence (2020, methodology): “Several new LegalAI datasets have been proposed (Kano et al., 2018; Xiao et al., 2018; Duan et al., 2019; Chalkidis et al., 2019b,a), which can serve as benchmarks for research in the field.”
• Combining Similarity and Transformer Methods for Case Law Entailment (2019, methodology): “The �rst COLIEE edition which included this task was in 2018 [18], and the two best performing approaches are described below.”