GPT-4 passes the bar exam

By Daniel Martin Katz, Michael James Bommarito II, Shang Gao, Pablo Arredondo. Philosophical Transactions of the Royal Society A, 2024

640 citations recorded by the source corpus.

In this paper, we experimentally evaluate the zero-shot performance of GPT-4 against prior generations of GPT on the entire uniform bar examination (UBE), including not only the multiple-choice multistate bar examination (MBE), but also the open-ended multistate essay exam (MEE) and multistate performance test (MPT) components. On the MBE, GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. On the MEE and MPT, which have not previously been evaluated by scholars, GPT-4 scores an average of 4.2/6.0 when compared with much lower scores for ChatGPT. Graded across the UBE components, in the manner in which a human test-taker would be, GPT-4 scores approximately 297 points, significantly in excess of the passing threshold for all UBE jurisdictions. These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society. This article is part of the theme issue ‘A complexity science approach to law and governance’.

Subfields: Artificial Intelligence, Political Science and International Relations

View Paper: https://www.semanticscholar.org/paper/0e8297549b4ec852ce8fd55dee9ae21501805af2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Political Science and International Relations

Citations in the corpus

The promise and challenges of generative AI in education (2024, neutral): “Notably, advanced LLMs have sur-passed most law school graduates on the bar exam (Katz et al. 2023), successfully completed medical, law, and business exams (Achiam et al. 2023), and have even excelled in a challenging US medical licensing exam, albeit not yet at the level of human doctors (Brin et…”
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges (2024, background): “Significantly, it achieved a ranking inside the highest decile of participants on a simulated iteration of the Uniform Bar Examination [44].”
Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, neutral): “…that leverage LLMs are increasingly enhancing the efficiency of numerous human activities, such as coding (Chen et al., 2021; Gao et al., 2023b), medical assistance (Yang et al., 2022; Moor et al., 2023), education (Kasneci et al., 2023; Kung et al., 2023), law (Katz et al., 2023), and so forth.”
How is ChatGPT's behavior changing over time? (2023, methodology): “Researchers have evaluated temporal shifts of previous (non-LLM) AI models (L. Chen et al., 2021).”
Causal Reasoning and Large Language Models: Opening a New Frontier for Causality (2023, background): “…complex, emergent behaviors, such as abilities to write code in programming languages, generate stories, poems, essays, and other texts, and demonstrate strong performance in certain reasoning tasks (Chen et al., 2021; Nguyen & Nadi, 2022; Bubeck et al., 2023; Katz et al., 2023; Wei et al., 2022a).”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “As a result, when confronted with problems necessitating domain-specific knowledge, such as medical (Li et al., 2023g; Singhal et al., 2023) and legal (Yu et al., 2022; Katz et al., 2023) questions, these models may exhibit pronounced hallucinations, often manifesting as factual fabrication.”
GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions (2023, background): “GPT model has the potential to be utilized for a variety of activities, including the creation of images and videos in addition to its excellent language processing capabilities [49].”
BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (2023, methodology): “In fact, OpenAI’s GPT-4 technical report disclosed their use of safety-relevant RLHF training prompts and rule-based reward models (RBRMs) [11] to empower safety alignment.”
Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models (2023, background): “While UQ has been an important topic for classical machine learning tasks like classification or regression (Lakshminarayanan et al., 2017; Gal & Ghahramani, 2016; Hern´andez-Lobato & Adams, 2015; Abdar et al., 2021a), it bears specific challenges for NLG and has attracted limited attention until re”
Building Legal Reward Models for Grounding and Abstention (2026, unknown): “Answers rarely reduce to purely deductive reasoning, instead requiring contextual interpretation and the weighing of competing legal considerations (Guha et al., 2023; Katz et al., 2024; Pipitone and Alami, 2024; Fan et al., 2026).”
Statistical inference on black-box generative models in the data kernel perspective space (2024, unknown): “Generative models have recently met or surpassed human-level standards on benchmarks across a range of tasks (Nori et al., 2023; Katz et al., 2024; Dubey et al., 2024).”