JEC-QA: A Legal-Domain Question Answering Dataset

By Haoxiang Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, Maosong Sun. 2019

220 citations recorded by the source corpus.

We present JEC-QA, the largest question answering dataset in the legal domain, collected from the National Judicial Examination of China. The examination is a comprehensive evaluation of professional skills for legal practitioners. College students are required to pass the examination to be certified as a lawyer or a judge. The dataset is challenging for existing question answering methods, because both retrieving relevant materials and answering questions require the ability of logic reasoning. Due to the high demand of multiple reasoning abilities to answer legal questions, the state-of-the-art models can only achieve about 28% accuracy on JEC-QA, while skilled humans and unskilled humans can reach 81% and 64% accuracy respectively, which indicates a huge gap between humans and machines on this task. We will release JEC-QA and our baselines to help improve the reasoning ability of machine comprehension models. You can access the dataset from http://jecqa.thunlp.org/.

Subfields: Artificial Intelligence, Political Science and International Relations, Evaluation & Benchmarks, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Political Science and International Relations

In Evaluation & Benchmarks

In Question Answering & Reading Comprehension

Citations in the corpus

AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, positive): “For LogiQA, JEC-QA and LSAT, we use ChatGPT to generate explanations given the questions and the answers.”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “Subsequently, fine-tuning occurs through the analysis of objective questions from the Chinese National Judicial Examination (Zhong et al., 2020) and the gathering of responses to legal consultations using ChatGPT.”
s1: Simple test-time scaling (2025, unknown): “…Biology, Chemistry, Computer Science, Geography, Math, and Physics olympiad questions 4250 3.2K OmniMath (Gao et al., 2024a) Math problems from competitions 4238 4.4K AGIEval (Zhong et al., 2023; Ling et al., 2017; Hendrycks et al., 2021; Liu et al., 2020; Zhong et al., 2019; Wang et al.,”
A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering (2020, unknown): “This work is concerned with the problem of statutory reasoning [62, 66]: how to reason about an example situation, a case, based on complex rules provided in natural language.”
Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents (2021, unknown): “…attempt to achieve performance gain by domain-adaptive pre-training on various domains, including the biomedical domain (Lee et al., 2020), the clinical domain (Alsentzer et al., 2019), the sci-entiﬁc domain (Beltagy et al., 2019), and the legal domain (Zhong et al., 2019; Chalkidis et al., 2020).”
How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence (2020, methodology): “JEC-QA contains 28, 641 multiple-choice and multiple-answer questions, together with 79, 433 relevant articles to help to answer the questions.”
When does pretraining help?: assessing self-supervised learning for law and the CaseHOLD dataset of 53,000+ legal holdings (2021, background): “This is explained in part due to the expense of labeling decisions and challenges around compiling large sets of legal documents [32], leading approaches above to rely on non-English datasets [49, 50] or proprietary datasets [14].”
Cross-Policy Compliance Detection via Question Answering (2021, background): “, 2020), enforcing community guidelines on social media platforms (Waseem and Hovy, 2016) or the resolution of legal cases (Zhong et al., 2020), etc.”
Baichuan 2: Open Large-scale Language Models (2023, methodology): “, 2021) and JEC-QA (Zhong et al., 2020), Baichuan 2 outperforms other opensource models, making it a suitable foundation model for domain-specific optimization.”