ChatGPT Goes to Law School

By Jonathan H. Choi, Kristin E. Hickman, Amy B. Monahan, D. Schwarcz. 2023

How well can AI models write law school exams without human assistance? To find out

Subfields: Large Language Models, Social Sciences & Humanities

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Social Sciences & Humanities

Citations in the corpus

AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, neutral): “Lawyer Qualification Test: Including the bar exam, these tests assess legal knowledge, analytical skills, and ethical understanding.”
Promises and challenges of generative artificial intelligence for human learning (2024, unknown): “Texttoimage models such as Stable Diffusion, Midjourney and DALLE 26,27 enable educators to create visual learning materials from textual content.”
Faith and Fate: Limits of Transformers on Compositionality (2023, background): “” – Charles Dickens, A Tale of Two Cities Large-scale Transformers such as ChatGPT [40] and GPT-4 [41] demonstrate unprecedented capabilities [40, 56, 8, 11, 65], even noted as “sparks of AGI” [9].”
On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective (2023, background): “There are reflections and discussions from law (Choi et al., 2023), education (Khalil and Er, 2023; M Alshater, 2022; Susnjak, 2022; Guo et al.”
A Survey of Large Language Models (2023, unknown): “, legal document analysis [409, 410], legal judgment prediction [411], and legal document writing [412].”
Gender bias and stereotypes in Large Language Models (2023, background): “In the context in which LLM-based applications have been either proposed or developed for real-world applications in the domains of medicine [12, 38, 68, 79], law [3, 23, 73, 74], finance [15, 26, 99], education [17, 62, 79], and many others, the potential repercussions of reproducing and amplifying”
Uncovering ChatGPT’s Capabilities in Recommender Systems (2023, background): “These LLMs possess a versatile nature and extensive world knowledge, allowing them to be applied not only in various NLP tasks [3, 5, 16], but also in domains such as education [11, 19, 22], medicine [1, 2, 29], search [20, 26, 37] and law [4, 8, 38].”
AI literacy and its implications for prompt engineering strategies (2024, background): “…high-quality and convincing output has generated excitement among students in higher education because it could be used to write essays and assignments (Dwivedi et al., 2023) and outscore human counterparts in a variety of domains (such as Law, e.g., Choi, Hickman, Monahan, & Schwarcz, 2023).”
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “, in education (Susnjak, 2022; Khalil and Er, 2023), law (Choi et al., 2023), medical (Kung et al.”
Performance of ChatGPT, GPT-4, and Google Bard on a Neurosurgery Oral Boards Preparation Question Bank (2023, background): “8 The goals of this study were to (1) assess the performance of 3 LLMs on a question bank with higher-order questions more representative of oral board topics and (2) elucidate differences in accuracy and performance by question characteristics between LLMs.”
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (2023, methodology): “The use of ChatGPT in Law School has also been explored [34].”
Welcome to the Era of ChatGPT et al. (2023, background): “…today’s knowledge work, ChatGPT has already passed a Wharton MBA exam (Mollman 2023), and seems to do fairly well in the US Medical Licensing Exam (Kung et al. 2022) as well as in (at least the multiple-choice sections of) judicial qualiﬁ-cation exams (Bommarito and Katz 2022; Choi et al. 2023).”
Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models (2023, background): “strated in increasingly diverse areas [24, 25], it is essential to address the biases that may exist within these models and their potential implications on users and society.”