Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

By Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso and 445 more. 2022

2,561 citations recorded by the source corpus.

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit"breakthrough"behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.

Subfields: Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, neutral): “We analysed Chinchilla on the same set of BIG-bench tasks (BIG-bench collaboration, 2021) reported in Rae et al. (2021).”
PaLM: Scaling Language Modeling with Pathways (2022, methodology): “Specifically, we evaluate the models on English Natural Language Processing (NLP) tasks (Section 6.1), tasks from BIG-bench (BIG-bench collaboration, 2021), reasoning tasks (Section 6.3), code completion tasks (Section 6.4), multilingual generation and question answering tasks (Section 6.6),…”
Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “We then ranked each pair based on helpfulness, honesty, and harmlessness, resulting in 221 binary comparisons [Srivastava et al., 2022].”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, background): “Although language models have demonstrated remarkable success across a range of NLP tasks, their ability to demonstrate reasoning is often seen as a limitation, which cannot be overcome solely by increasing model scale (Rae et al., 2021; BIG-bench collaboration, 2021, inter alia ).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “This has given rise to extremely broad capabilities, and models perform extremely well on a wide set of tasks (Srivastava et al., 2022; Liang et al., 2022).”
Program Synthesis with Large Language Models (2021, background): “P L ] 1 6 A ug 2 text [Brown et al., 2020, Raffel et al., 2019] and to solve a rapidly expanding set of modeling and reasoning tasks [Devlin et al., 2019, big-bench collaboration, 2021].”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “, 2021a), and Big Bench Hard (BBH) (Srivastava et al., 2022).”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “Both of these approaches are promising, and have now demonstrated competitive results–at large scales for MoE (Fedus et al., 2022; Srivastava et al., 2022), and at smaller scale for state-space models with H3 (Anonymous, 2023).”
Augmented Language Models: a Survey (2023, background): “, 2022) use chain-of-thought style prompts to achieve remarkable improvements on popular benchmarks such as BBH (Srivastava et al., 2022) and MMLU (Hendrycks et al.”
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, methodology): “Suzgun, M., Scales, N., Scharli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., ZHou, D., and Wei, J. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.”
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, methodology): “BIG-bench: Hard The “hard” subset of BIGbench (Suzgun et al., 2022) contains 23 challenging tasks from BIG-Bench (Srivastava et al., 2022).”
Large Language Models Are Human-Level Prompt Engineers (2022, background): “These so-called “large language models” (LLMs) have shown remarkable, often superhuman, capabilities across a diverse range of tasks, including both zero-shot and few-shot setups (Brown et al., 2020; Srivastava et al., 2022).”
Scaling Instruction-Finetuned Language Models (2022, methodology): “…we evaluate both the ability to directly predict the answer via direct prompting, where the model directly gives the answer (Brown et al., 2020; Srivastava et al., 2022), as well as via chain-of-thought (CoT) prompting, where the model must provide a reasoning chain before giving the ﬁnal…”
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (2022, methodology): “While answer-only prompting has been used as the stan-4 dard in many prior work (Brown et al., 2020; Rae et al., 2021; Hoffmann et al., 2022; Srivastava et al., 2022), it typically underestimates model performance on challenging tasks, such as those that require multiple reasoning steps.”
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “With the recent advances of LLMs, LLM-based assistants start to exhibit artificial general intelligence across diverse tasks, from writing and chatting to coding [5, 26, 1, 32].”
A Review of Sparse Expert Models in Deep Learning (2022, methodology): “The ﬁeld is still uncovering properties of sparse expert models, including much improved calibration (Srivastava et al., 2022); others remain unknown including their dynamics under asynchronous training (Recht et al., 2011) or their memorization abilities (Carlini et al., 2020).”