AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

By Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, A. Saied, Weizhu Chen, Nan Duan. 2023

867 citations recorded by the source corpus.

Evaluating the general abilities of foundation models to tackle human-level tasks is a vital aspect of their development and application in the pursuit of Artificial General Intelligence (AGI). Traditional benchmarks, which rely on artificial datasets, may not accurately represent human-level capabilities. In this paper, we introduce AGIEval, a novel benchmark specifically designed to assess foundation model in the context of human-centric standardized exams, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests. We evaluate several state-of-the-art foundation models, including GPT-4, ChatGPT, and Text-Davinci-003, using this benchmark. Impressively, GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, attaining a 95% accuracy rate on the SAT Math test and a 92.5% accuracy on the English test of the Chinese national college entrance exam. This demonstrates the extraordinary performance of contemporary foundation models. In contrast, we also find that GPT-4 is less proficient in tasks that require complex reasoning or specific domain knowledge. Our comprehensive analyses of model capabilities (understanding, knowledge, reasoning, and calculation) reveal these models' strengths and limitations, providing valuable insights into future directions for enhancing their general capabilities. By concentrating on tasks pertinent to human cognition and decision-making, our benchmark delivers a more meaningful and robust evaluation of foundation models' performance in real-world scenarios. The data, code, and all model outputs are released in https://github.com/ruixiangcui/AGIEval.

Subfields: Artificial Intelligence, Health Informatics, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Evaluation & Benchmarks

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “In Table 24, we present fine-grained results from the English part of the AGI Eval (Zhong et al., 2023) benchmark.”
Mistral 7B (2023, background): “…with maj@4 • Code: Humaneval [5] (0-shot) and MBPP [2] (3-shot) • Popular aggregated results: MMLU [12] (5-shot), BBH [23] (3-shot), and AGI Eval [29] (3-5-shot, English multiple-choice questions only) Detailed results for Mistral 7B, Llama 2 7B/13B, and Code-Llama 7B are reported in Table 2.”
Mixtral of Experts (2024, background): “…with maj@4 • Code: Humaneval [4] (0-shot) and MBPP [1] (3-shot) • Popular aggregated results: MMLU [16] (5-shot), BBH [29] (3-shot), and AGI Eval [34] (3-5-shot, English multiple-choice questions only) Figure 2: Performance of Mixtral and different Llama models on a wide range of benchmarks .”
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, methodology): “• Core-knowledge benchmarks, including MMLU [16], HellaSwag [41], ARC [8], WinoGrande [31], HumanEval [6], GSM-8K [9], and AGIEval [42], evaluate the core capabilities of pre-trained LLMs using zero-shot and few-shot benchmark sets.”
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, background): “Common ones include MMLU [16], MATH [17], GSM-8K [10], HumanEval [7], DROP [12], BigBench [37], Hel-laSwag [49], AGIEval [52], GPQA [35], as well as comprehensive collection like HELM [26].”
MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, background): “For example, AGIEval [48] focuses on general-exam questions from SAT, Gaokao, GRE, etc. ARC [12] focuses on science-based questions.”
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “Chinese benchmarks include MGSM-zh (Shi et al., 2023), CMATH (Wei et al., 2023), Gaokao-MathCloze (Zhong et al., 2023), and Gaokao-MathQA (Zhong et al., 2023).”
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models (2023, methodology): “Recently, the AGIEval benchmark (Zhong et al., 2023) contains data from the Chinese College Entrance Exam, Chinese lawyer qualification test and Chinese civil service examination.”
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, background): “However, existing benchmarks (Hendrycks et al., 2020; Zhong et al., 2023; Zheng et al., 2023a) for evaluating LLMs mainly focus on NLP tasks, such as common sense reasoning, academic examination, and authenticity verification.”
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism (2024, neutral): “We apply generation-based evaluation for TriviaQA, NaturalQuestions, DROP, MATH, GSM8K, HumanEval, MBPP, BBH, AGIEval, CLUEWSC, and CMath.”
CMMLU: Measuring massive multitask language understanding in Chinese (2023, background): “AGIEval (Zhong et al., 2023) focuses on Chinese standardized exams, such as college entrance exams, math competitions, and lawyer qualiﬁcation tests.”
When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards (2024, neutral): “Leaderboards based on multiple choice questions (MCQ) for evaluation (Wang et al., 2018, 2019; Nie et al., 2019; Zhong et al., 2023; Hendrycks et al., 2020) present both convenience and significant limitations (Pezeshkpour and Hr-uschka, 2023; Zheng et al., 2023).”
Don't Make Your LLM an Evaluation Benchmark Cheater (2023, background): “Recently, a surge of high-quality evaluation benchmarks (Hendrycks et al., 2021; Huang et al., 2023) have been proposed to provide a comprehensive capability evaluation of LLMs. Typical benchmarks include MMLU (Hendrycks et al., 2021) (for measuring multitask language understanding ability), Big-Ben”
MathPile: A Billion-Token-Scale Pretraining Corpus for Math (2023, neutral): “Here, we utilize popular mathematical reasoning benchmarks, namely GSM8K [16], MATH [26], MMLU-STEM [25], AGIEval-SAT-MATH [70], MathQA [3] and AQuA [39] to detect data contamination.”
The Lessons of Developing Process Reward Models in Mathematical Reasoning (2025, neutral): “We evaluate across three Chinese benchmarks including Chinese math benchmarks CMATH (Wei et al., 2023), GaoKao Math Cloze (Zhong et al., 2024), and GaoKao Math QA (Zhong et al., 2024) following Yang et al. (2024c), as shown in (Liao et al., 2024) and OlympiadBench (He et al., 2024).”
Causal Reasoning and Large Language Models: Opening a New Frontier for Causality (2023, background): “Now, LLMs may have the capability to automate or assist with every step of this process and seamlessly transition bteween covariance-and logic-based causality. soning (Bubeck et al., 2023; Zhong et al., 2023; Ghazal et al., 2017).”
L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning (2025, neutral): “We evaluate our models on test sets of 4 different reasoning datasets: AIME 2025, MATH (Hendrycks et al., 2021b), AMC, Olympiad-Bench (He et al., 2024), and additionally GPQA (Rein et al., 2023), LSAT (Zhong et al., 2023), and MMLU (Hendrycks et al., 2021a).”