Meta-learning via Language Model In-context Tuning

By Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, He He. 2021

208 citations recorded by the source corpus.

The goal of meta-learning is to learn to adapt to a new task with only a few labeled examples. Inspired by the recent progress in large language models, we propose \textit{in-context tuning} (ICT), which recasts task adaptation and prediction as a simple sequence prediction problem: to form the input sequence, we concatenate the task instruction, labeled in-context examples, and the target input to predict; to meta-train the model to learn from in-context examples, we fine-tune a pre-trained language model (LM) to predict the target label given the input sequence on a collection of tasks.We benchmark our method on two collections of text classification tasks: LAMA and BinaryClfs. Compared to MAML which adapts the model through gradient descent, our method leverages the inductive bias of pre-trained LMs to perform pattern matching, and outperforms MAML by an absolute 6% average AUC-ROC score on BinaryClfs, gaining more advantage with increasing model size. Compared to non-fine-tuned in-context learning (i.e. prompting a raw LM), in-context tuning meta-trains the model to learn from in-context examples. On BinaryClfs, ICT improves the average AUC-ROC score by an absolute 10%, and reduces the variance due to example ordering by 6x and example choices by 2x.

Subfields: Artificial Intelligence, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, background): “…for the demonstrations (Liu et al., 2021; Lu et al., 2021; Rubin et al., 2021), meta-training with an explicit in-context learning objective (Chen et al., 2021; Min et al., 2021b), and learning to follow instructions as a variant of in-context learning (Mishra et al., 2021b; Efrat and Levy,…”
Learning by Distilling Context (2022, background): “…works show that language models can learn from abstract task deﬁnitions (Zhong et al., 2021; Mishra et al., 2022; Wei et al., 2022a; Sanh et al., 2022), natural language explanations (Scheurer et al., 2022; Wang et al., 2022b), and concrete in-context examples (Min et al., 2022; Chen et al., 2022).”
MetaICL: Learning to Learn In Context (2021, methodology): “Elimination of templates Prior work uses human-authored templates to transform the inputoutput pair to a natural language sentence (Mishra et al., 2021b; Zhong et al., 2021; Wei et al., 2021; Chen et al., 2021).”
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model (2022, methodology): “Multi-task learning approaches, which explicitly finetune on the in-context learning objective by using numerous NLP tasks, are proposed recently to tackle zero/few-shot transfer to the unseen task at test time (Wei et al., 2021; Sanh et al., 2021; Chen et al., 2021b; Min et al., 2021).”
Instruction Induction: From Few Examples to Natural Language Task Descriptions (2022, methodology): “[2021] study methods for improving the stability and accuracy of prompt-based models, Chen et al. [2021] and Min et al. [2022a] conduct meta-training with an in-context learning objective, while other work studies the effect of the provided prompts [Reynolds and McDonell, 2021, Webson and Pavlick, 2”
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes (2022, background): “Since Brown et al. [10] demonstrated the in-context learning ability of GPT-3, there has been a significant interest in improving and understanding this capability [36, 39, 79, 38, 59, 40, 14, 43, 33].”
Selective Annotation Makes Language Models Better Few-Shot Learners (2022, methodology): “Several recent works proposed methods to improve in-context learning in many aspects: e.g., meta-training (Chen et al., 2022; Min et al., 2022b), task instructions (Efrat & Levy, 2020; Mishra et al., 2022; Wei et al., 2021; Sanh et al., 2022), or task formulation (Holtzman et al., 2021; Zhao et…”
A Survey on In-context Learning (2022, methodology): “While the vanilla GPT-3 model itself shows promising ICL abilities, several studies observed that the ability could be significantly boosted via adaption during pretraining (Min et al., 2022b; Chen et al., 2022c).”
A Survey for In-context Learning (2023, methodology): “Unlike traditional optimization-based meta-learning approaches, Chen et al. (2022d) applied ICL for meta-learning.”
PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “Our work is aligned in spirit to Flamingo [10] that uses interleaved image text pairs in the same web page and in-context tuning [18] during pre-training.”
UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training (2022, background): “The past few years has seen a lot of activity toward unified model designs (Friedman et al., 2021; Aghajanyan et al., 2021; Gupta et al., 2021; Tafjord and Clark, 2021; Lourie et al., 2021; Jiang et al., 2021; Aribandi et al., 2021; Zhong et al., 2021a; Chen et al., 2021; Zhong et al., 2022).”
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (2023, background): “Large Language Models (LLMs) [7, 16, 42, 59, 60, 65, 69] have achieved great success in various natural language processing tasks, e.g. , topic classification [31, 33, 47], sentiment classification [7, 47], translation [7], by few-shot prompting (or in-context learning) [7, 10, 47].”
Successive Prompting for Decomposing Complex Questions (2022, background): “Second, when ﬁne-tuning (with or without in-context examples (Chen et al., 2022)), we can provide training examples for each step independently, so the model only has to learn to perform one step at a time.”
What learning algorithm is in-context learning? Investigations with linear models (2022, background): “This behavior occurs both in models trained on collections of few-shot learning problems (Chen et al., 2022; Min et al., 2022) and surprisingly in large language models trained on open-domain text (Brown et al., 2020; Zhang et al., 2022; Chowdhery et al., 2022).”