Quantifying Adaptability in Pre-trained Language Models with 500 Tasks

By Belinda Z. Li, Jane A. Yu, Madian Khabsa, Luke S. Zettlemoyer, Alon Y. Halevy, Jacob Andreas. 2021

18 citations recorded by the source corpus.

When a neural language model (LM) is adapted to perform a new task, what aspects of the task predict the eventual performance of the model? In NLP, systematic features of LM generalization to individual examples are well characterized, but systematic aspects of LM adaptability to new tasks are not nearly as well understood. We present a large-scale empirical study of the features and limits of LM adaptability using a new benchmark, TaskBench500, built from 500 procedurally generated sequence modeling tasks. These tasks combine core aspects of language processing, including lexical semantics, sequence processing, memorization, logical reasoning, and world knowledge. Using TaskBench500, we evaluate three facets of adaptability, finding that: (1) adaptation procedures differ dramatically in their ability to memorize small datasets; (2) within a subset of task types, adaptation procedures exhibit compositional adaptability to complex tasks; and (3) failure to match training label distributions is explained by mismatches in the intrinsic difficulty of predicting individual labels. Our experiments show that adaptability to new tasks, like generalization to new examples, can be systematically described and understood, and we conclude with a discussion of additional aspects of adaptability that could be studied using the new benchmark.

Subfields: Artificial Intelligence, Large Language Models, Evaluation & Benchmarks, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ekin Akyürek
2. Alexis Ross
3. Dimitris Papailiopoulos
4. Jy-yong Sohn
5. Bailin Wang
6. Michael Gira
7. Ruisu Zhang
8. Shashank Rajput

In Large Language Models

1. Jacob Andreas
2. Kangwook Lee
3. Michael Gira
4. Ruisu Zhang
5. Najoung Kim

In Evaluation & Benchmarks

1. Dimitris Papailiopoulos
2. Jacob Andreas
3. Kangwook Lee
4. Alexis Ross
5. Najoung Kim

In Meta & Continual Learning

1. Ekin Akyürek
2. Jacob Andreas

Citations in the corpus

• LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “Analyzing the adaptability of LMs. Similar to ours, recent works [81, 82, 83, 82] attempt to understand and quantify the adaptability [20] and capacity of large LMs, such as Big-Bench [82] with a new benchmark of more than 200 tasks on a diverse set of topics.”
• Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, background): “In this work, we are interested in the models’ generalizability to new task variants, which has been less systematically studied for LMs (though see Li et al. (2022), Mishra et al. (2022), and Wang et al. (2022b)).”