Holistic Evaluation of Language Models

By Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, D. Narayanan, Yuhuai Wu, Ananya Kumar and 42 more. 2023

1,863 citations recorded by the source corpus.

Language models (LMs) like GPT‐3, PaLM, and ChatGPT are the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of LMs. LMs can serve many purposes and their behavior should satisfy many desiderata. To navigate the vast space of potential scenarios and metrics, we taxonomize the space and select representative subsets. We evaluate models on 16 core scenarios and 7 metrics, exposing important trade‐offs. We supplement our core evaluation with seven targeted evaluations to deeply analyze specific aspects (including world knowledge, reasoning, regurgitation of copyrighted content, and generation of disinformation). We benchmark 30 LMs, from OpenAI, Microsoft, Google, Meta, Cohere, AI21 Labs, and others. Prior to HELM, models were evaluated on just 17.9% of the core HELM scenarios, with some prominent models not sharing a single scenario in common. We improve this to 96.0%: all 30 models are now benchmarked under the same standardized conditions. Our evaluation surfaces 25 top‐level findings. For full transparency, we release all raw model prompts and completions publicly. HELM is a living benchmark for the community, continuously updated with new scenarios, metrics, and models https://crfm.stanford.edu/helm/latest/.

Subfields: Artificial Intelligence, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “We evaluate on the test sets for MMLU and BBH and on the evaluation split for RAFT released by the HELM benchmark (Liang et al., 2022).”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, methodology): “Evaluation suites across system characteristics for specific generative system modalities, such as language, include Holistic Evaluation of Language Models (HELM) [139], BigBench [232], Language Model Evaluation Harness [85].”
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, neutral): “The following categories of human-centric exams are included in our benchmark: General College Entrance Exams: Including the GRE, SAT, and Gaokao, these exams assess critical thinking, problem-solving, and analytical skills for entry into higher education.”
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “However, the heightened user preference does not always correspond to improved scores on traditional LLM benchmarks – benchmarks like MMLU [16] and HELM [20] cannot effectively tell the difference between these aligned models and the base models.”
QLoRA: Efficient Finetuning of Quantized LLMs (2023, background): “Math Guanaco’s biggest weakness is mathematics, an area where many language models struggle in [35].”
StarCoder: may the source be with you! (2023, methodology): “We evaluate the ability of the model to leverage its natural language and code pretraining for natural language reasoning tasks from HELM (excluding code tasks, because of our own extensive code evaluations).”
Foundation Models and Fair Use (2023, methodology): “In contrast, most technical work on copyright evaluation and mitigation focuses on near-verbatim overlap, which we argue is insufficient on its own (Ziegler, 2021; Liang et al., 2022; Vyas et al., 2023).”
The Gradient of Generative AI Release: Methods and Considerations (2023, background): “These figures are based on tracking and evaluation initiatives [49, 50, 94], are not exhaustive, and intend to show release trends over”
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, background): “Common ones include MMLU [16], MATH [17], GSM-8K [10], HumanEval [7], DROP [12], BigBench [37], Hel-laSwag [49], AGIEval [52], GPQA [35], as well as comprehensive collection like HELM [26].”
Are We Done with MMLU? (2024, neutral): “Although our analysis uses a significant subset of 3,000 questions, the accuracy of this analysis, both quantitatively and qualitatively, can be further improved with additional annotation of the remaining 12,908 questions in MMLU.”
MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, background): “Other recent benchmarks, including MMLU [18], HELM [22], BigBench [32], Hel-laSwag [45], and the AI2 Reasoning Challenge (ARC) [12], have broadened the scope by assessing capabilities across language generation, knowledge understanding, and complex reasoning [9].”
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models (2023, background): “, 2022), and HELM (Liang et al., 2022) benchmarks attempt to aggregate a wide range of NLP tasks for holistic evaluation.”
Large Language Models (2023, background): “23 The benchmark suite HELM [80] measures additional metrics such as tendency to repeat copyrighted material, bias, toxicity and the like.”
Inverse Scaling: When Bigger Isn't Better (2023, background): “HELM (Liang et al., 2022) is a living benchmark intended to holistically evaluate the capabilities and limitations LMs.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “Salient documentation proposals focus on regular reporting of reward components [Gilbert et al., 2022] and the ability to compare the capabilities of language models according to standard benchmarks [Liang et al., 2022a].”