MetaICL: Learning to Learn In Context

By Sewon Min, Michael Lewis, Luke S. Zettlemoyer, Hannaneh Hajishirzi. 2021

627 citations recorded by the source corpus.

We introduce MetaICL (Meta-training for In-Context Learning), a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learning on a large set of training tasks. This meta-training enables the model to more effectively learn a new task in context at test time, by simply conditioning on a few training examples with no parameter updates or task-specific templates. We experiment on a large, diverse collection of tasks consisting of 142 NLP datasets including classification, question answering, natural language inference, paraphrase detection and more, across seven different meta-training/target splits. MetaICL outperforms a range of baselines including in-context learning without meta-training and multi-task learning followed by zero-shot transfer. We find that the gains are particularly significant for target tasks that have domain shifts from the meta-training tasks, and that using a diverse set of the meta-training tasks is key to improvements. We also show that MetaICL approaches (and sometimes beats) the performance of models fully finetuned on the target task training data, and outperforms much bigger models with nearly 8x parameters.

Subfields: Artificial Intelligence, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

Citations in the corpus

Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “Perhaps the papers most related to ours are the recent Sanh et al. (2021) and Min et al. (2021), which were released after our initial preprint.”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “We present validation set results for instructiontuning with different settings for MetaICL, aggregated by generalization level and task category under both 0 and 5-shot settings, in Table 8.”
OPT: Open Pre-trained Transformer Language Models (2022, background): “Recent efforts have shown gains by ﬁne-tuning models to directly respond to instruction-style prompting (Wei et al., 2021; Min et al., 2021; Sanh et al., 2021; Ouyang et al., 2022).”
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, methodology): “MetaICL (Min et al., 2022) also consolidated other task collections (Ye et al.”
Scaling Instruction-Finetuned Language Models (2022, methodology): “This degradation may be surprising in light of multiple prior studies ﬁnding that instruction ﬁnetuning improves performance on unseen tasks (Wei et al., 2021; Sanh et al., 2021; Wang et al., 2019a; Min et al., 2022, inter alia ).”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, background): “C L ] 20 O c t 2022 (3) meta-training with an in-context learning objec-tive (Min et al., 2021b) magniﬁes these effects—the models almost exclusively exploit simpler aspects of the demonstrations like the format rather than the input-label mapping.”
Learning by Distilling Context (2022, background): “…works show that language models can learn from abstract task deﬁnitions (Zhong et al., 2021; Mishra et al., 2022; Wei et al., 2022a; Sanh et al., 2022), natural language explanations (Scheurer et al., 2022; Wang et al., 2022b), and concrete in-context examples (Min et al., 2022; Chen et al., 2022).”
QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “Approaches and datasets include MetaICL [40], MetaTuning [74], InstructGPT [43], FLAN [63, 12], PromptSource [3], Super-NaturalInstructions [62, 51], Self-instruct [60], UnnaturalInstructions [26], OPT-IML [29], UnifiedSKG[68], OIG/Chip2 [32], Alpaca [56], Vicuna [10], Koala [20], and…”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “We compare against zero-shot learning with T0 [1] (since we found few-shot ICL to performed worse than zeroshot for T0, see appendix E); few-shot ICL with T5+LM [14] (the next-step-prediction language model upon which T0 is based); and few-shot ICL with the 6.7, 13, and 175 billion parameter variant”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, background): “, 2022) or demonstration examples (Min et al., 2022a).”
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts (2022, background): “Since then, PromptSource and P3 have been extended for research on multi-lingual prompting (Lin et al., 2021) and priming, i.e., in-context few-shot learning (Min et al., 2021).”
Poisoning Language Models During Instruction Tuning (2023, methodology): “Instruction-tuned Language Models The de-facto standard method for building state-of-the-art large LMs is via “instruction” or “meta” ﬁnetuning (Ouyang et al., 2022; Chung et al., 2022; Min et al., 2022; Zhong et al., 2021, inter alia ).”
Meta-learning via Language Model In-context Tuning (2021, background): “Contrary to our paper, Min et al. (2021) ﬁnds that in-context tuning under-performs InsT + FT.”
Specializing Smaller Language Models towards Multi-Step Reasoning (2023, result): “The reason we prepend the in-context example is that previous work shows tuning with in-context examples improves the model’s in-context learning ability (Min et al., 2022).”
Instruction Induction: From Few Examples to Natural Language Task Descriptions (2022, background): “…in-context learning, Zhao et al. [2021] study methods for improving the stability and accuracy of prompt-based models, Chen et al. [2021] and Min et al. [2022a] conduct meta-training with an in-context learning objective, while other work studies the effect of the provided prompts [Reynolds…”
Discovering Latent Knowledge in Language Models Without Supervision (2022, methodology): “Many methods improve upon vanilla zero-shot prompting (Liu et al., 2022; Zhao et al., 2021; Lu et al., 2022; Wei et al., 2022b; Min et al., 2022a).”
Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (2022, background): “Several works use intermediate adaptation to create a model that is better in the zero- or few-shot settings (Wei et al., 2022; Sanh et al., 2022; Min et al., 2022), but these target generalization to new tasks, whereas we focus on generalizing to new languages within one task.”