Measuring Massive Multitask Language Understanding
By Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt. 2020
8,927 citations recorded by the source corpus.
We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.
Subfields: Artificial Intelligence, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Jianfeng Gao
- Yejin Choi
- Quoc V. Le
- Graham Neubig
- Noah A. Smith
In Evaluation & Benchmarks
- Mohit Bansal
- Yejin Choi
- Hannaneh Hajishirzi
- Samuel R. Bowman
- Luke S. Zettlemoyer
- Douwe Kiela
- Ashish Sabharwal
- Jianfeng Gao
Citations in the corpus
- Training Compute-Optimal Large Language Models (2022, neutral): “In Table 6, we report Chinchilla ’s average 5-shot performance on MMLU (the full breakdown of results is shown in Table A6).”
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “Table 32 shows the few-shot performance on the following benchmarks: average NLU and NLG score on English NLP tasks, average score on the MMLU benchmark using 5-shot prompts, average normalized preferred metric on 58 BIG-bench tasks (that are overlapping with Chinchilla) using 5-shot prompts, accura”
- LLaMA: Open and Efficient Foundation Language Models (2023, background): “The massive multitask language understanding benchmark, or MMLU, introduced by Hendrycks et al. (2020) consists of multiple choice questions covering various domains of knowledge, including humanities, STEM and social sciences.”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We report the overall results for MMLU (5 shot) (Hendrycks et al., 2020), Big Bench Hard (BBH) (3 shot) (Suzgun et al., 2022), and AGI Eval (3–5 shot) (Zhong et al., 2023).”
- Mistral 7B (2023, background): “…[10] (8-shot) with maj@8 and MATH [13] (4-shot) with maj@4 • Code: Humaneval [5] (0-shot) and MBPP [2] (3-shot) • Popular aggregated results: MMLU [12] (5-shot), BBH [23] (3-shot), and AGI Eval [29] (3-5-shot, English multiple-choice questions only) Detailed results for Mistral 7B, Llama 2…”
- Mixtral of Experts (2024, background): “…[9] (8-shot) with maj@8 and MATH [17] (4-shot) with maj@4 • Code: Humaneval [4] (0-shot) and MBPP [1] (3-shot) • Popular aggregated results: MMLU [16] (5-shot), BBH [29] (3-shot), and AGI Eval [34] (3-5-shot, English multiple-choice questions only) Figure 2: Performance of Mixtral and different…”
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “(Hendrycks et al., 2021).”
- TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “reason to focus on imitative falsehoods is that they are less likely to be covered by existing question-answering benchmarks (Clark et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017; Hendrycks et al., 2020).”
- Scaling Laws for Transfer (2021, background): “Another approach for measuring generalization we’re interested in is the development of increasingly difficult language benchmarks [HBB + 21].”
- Solving Quantitative Reasoning Problems with Language Models (2022, methodology): “When evaluating MMLU with majority voting, we sample k = 16 model answers using a chain-of-thought prompt.”
- Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm (2021, background): “For instance, its scores on amultitask dataset [10] barely exceed random guessing in many sections.”
- OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “Recently, along similar lines as this work, Chung et al. (2022b) achieve impressive gains on the challenging benchmarks of MMLU (Hendrycks et al., 2020) and Big-Bench Hard (Suzgun et al., 2022) by instruction-tuning PaLM (Chowdhery et al., 2022) and T5 (Raffel et al., 2020) on a scaled-up…”
- Emergent Abilities of Large Language Models (2022, positive): “Figure 2G shows the Massive Multi-task Language Understanding (MMLU) benchmark, which aggregates 57 tests covering a range of topics including math, history, law, and more (Hendrycks et al., 2021a).”
- GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “Hendrycks et al. (2021b) find that few-shot evaluation does not improve performance, but that appears to be only the case for GPT-3.”
- Measuring Mathematical Problem Solving With the MATH Dataset (2021, background): “Hendrycks et al. (2021) create a multiple-choice benchmark covering 57 subjects.”
- The False Promise of Imitating Proprietary LLMs (2023, methodology): “We evaluate using human and GPT-4 evaluations (blind pairwise comparisons with ChatGPT) as well as accuracy on canonical NLP benchmarks (MMLU, NQ, HumanEval).”
- AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, neutral): “(2) AGIEval is bilingual (English and Chinese), broadening the assessment scope across languages and cultures, whereas MMLU is solely English-based.”
- Augmented Language Models: a Survey (2023, background): “Further, several instruction fine-tuning approaches (Ouyang et al., 2022; Chung et al., 2022; Iyer et al., 2022; Ho et al., 2022) use chain-of-thought style prompts to achieve remarkable improvements on popular benchmarks such as BBH (Srivastava et al., 2022) and MMLU (Hendrycks et al., 2021).”
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, methodology): “As shown in Figure 1, this includes a 4.2%+ and 8.5% improvements on the MMLU (Hendrycks et al., 2020) and BIG-Bench Hard (Suzgun et al., 2022) evaluation benchmarks, for equally sized models.”
- Galactica: A Large Language Model for Science (2022, methodology): “The model outperforms GPT-3 on MMLU, a model 16 times larger.”