CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

By Qinyuan Ye, Bill Yuchen Lin, Xiang Ren. 2021

202 citations recorded by the source corpus.

Humans can learn a new language task efficiently with only few examples, by leveraging their knowledge obtained when learning prior tasks. In this paper, we explore whether and how such cross-task generalization ability can be acquired, and further applied to build better few-shot learners across diverse NLP tasks. We introduce CrossFit, a problem setup for studying cross-task generalization ability, which standardizes seen/unseen task partitions, data access during different learning stages, and the evaluation protocols. To instantiate different seen/unseen task partitions in CrossFit and facilitate in-depth analysis, we present the NLP Few-shot Gym, a repository of 160 diverse few-shot NLP tasks created from open-access NLP datasets and converted to a unified text-to-text format. Our analysis reveals that the few-shot learning ability on unseen tasks can be improved via an upstream learning stage using a set of seen tasks. We also observe that the selection of upstream learning tasks can significantly influence few-shot performance on unseen tasks, asking further analysis on task similarity and transferability.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Finetuned Language Models Are Zero-Shot Learners (2021, background): “This promising result (as well as one from Ye et al. (2021), which does not emphasize instructions as much) suggests that finetuning on a collection of tasks improves few-shot performance on unseen tasks, even at a smaller model scale.”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “, 2020), CrossFit (Ye et al., 2021); and areaspecific task consolidation: Structured Knowledge Grounding (Xie et al.”
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “MetaICL (Min et al., 2022) also consolidated other task collections (Ye et al., 2021; Khashabi et al., 2020) to train models to learn tasks “in-context” – from several input-output examples, known as few-shot prompting, but in this case without instructions.”
Scaling Instruction-Finetuned Language Models (2022, background): “…on either zero-shot without exemplars (Zhong et al., 2021; Wei et al., 2021; Sanh et al., 2021; Wang et al., 2022a) or few-shot with exemplars (Ye et al., 2021; Wei et al., 2021; Min et al., 2022; Wang et al., 2022c), our process ﬁnetunes on a mixture of both formats to enable both zero-shot…”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, background): “Multi-task learning is a long-standing goal for AI (Caruana, 1997) and has led to successful model that can support a wider range of tasks (McCann et al., 2018; Khashabi et al., 2020; Aghajanyan et al., 2021; Ye et al., 2021; Raffel et al., 2020).”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, background): “10 Future work can further improve the zero-shot performance with relaxed assumptions in access to the unlabeled training data.”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “, 2021) and fine-tuning PLMs on them, the PLMs acquire the cross-task transfer ability (Ye et al., 2021) to execute unseen tasks when they are formulated in a similar format.”
True Few-Shot Learning with Language Models (2021, neutral): “The meta-learning setting assumes access to data from many other tasks in order to perform learning and model selection [20, 21, 64, 65].”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, background): “, 2022b) CROSSFIT (Ye et al., 2021) PROMPTSOURCE (Bach et al.”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “U NIFIED SKG is a challenging testbed for few-shot (Brown et al., 2020; Ye et al., 2021a) and zero-shot learning (Zhong et al., 2021; Wei et al., 2021; Sanh et al., 2021) with PLMs.”
MetaICL: Learning to Learn In Context (2021, methodology): “All datasets are taken from CROSSFIT (Ye et al., 2021) (except we exclude datasets that are unavailable from their repository7 or the scope is notably different from other tasks, e.g., solving math problems or breaking down compositional questions) and UNIFIEDQA (Khashabi et al., 2020).”
ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, neutral): “For instance, Mishra et al. (Mishra et al., 2021) built a dataset with task instructions, and CROSSFIT (Ye et al., 2021) introduced a repository of few-shot text-to-text tasks.”
Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right (2021, background): “, 2020), an intermediate learning phase (Ye et al., 2021), or calibration (Zhao et al.”
FLEX: Unifying Evaluation for Few-Shot NLP (2021, methodology): “Very recent work has studied task transfer over a large set of datasets [72].”
Unsupervised Cross-Task Generalization via Retrieval Augmentation (2022, methodology): “Particularly, in the CrossFit framework (Ye et al., 2021), cross-task generalization requires a small number of labeled instances of the target task for ﬁne-tuning.”
LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition (2023, background): “Recent advancements like CrossFit (Ye et al., 2021), ExT5 (Aribandi et al., 2022), FLAN (Wei et al., 2022), T0 (Sanh et al., 2022), Instruct-GPT (Ouyang et al., 2022), and ReCross (Lin et al., 2022) have been striving to foster a vastly multi-task model’s generalization across different tasks, very…”
Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations (2022, methodology): “First, as a baseline template we used minimal template following (Ye et al., 2021; Min et al., 2022b).”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, methodology): “…studies pretrained language models’ crosstask generalization abilities for in-context fewshot or zero-shot learning using human-annotated datasets (Ye et al., 2021; Wei et al., 2022; Sanh et al., 2022; Min et al., 2021; Xu et al., 2022) via instructions (Weller et al., 2020; Efrat and Levy,…”