FLEX: Unifying Evaluation for Few-Shot NLP

By Jonathan Bragg, Arman Cohan, Kyle Lo, Iz Beltagy. 2021

113 citations recorded by the source corpus.

Few-shot NLP research is highly active, yet conducted in disjoint research threads with evaluation suites that lack challenging-yet-realistic testing setups and fail to employ careful experimental design. Consequently, the community does not know which techniques perform best or even if they outperform simple baselines. In response, we formulate the FLEX Principles, a set of requirements and best practices for unified, rigorous, valid, and cost-sensitive few-shot NLP evaluation. These principles include Sample Size Design, a novel approach to benchmark design that optimizes statistical accuracy and precision while keeping evaluation costs manageable. Following the principles, we release the FLEX benchmark, which includes four few-shot transfer settings, zero-shot evaluation, and a public leaderboard that covers diverse NLP tasks. In addition, we present UniFew, a prompt-based model for few-shot learning that unifies pretraining and finetuning prompt formats, eschewing complex machinery of recent prompt-based approaches in adapting downstream task formats to language model pretraining objectives. We demonstrate that despite simplicity, UniFew achieves results competitive with both popular meta-learning and prompt-based approaches.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

Citations in the corpus

Few-Shot Self-Rationalization with Natural Language Prompts (2021, methodology): “Prior work (Bragg et al., 2021) introduced U NI F EW , a model based on U NIFIED QA, that is ﬁnetuned on a few task-speciﬁc instances posed as QA.”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, background): “…been motivated by building models that are generalizable across a variety of NLP tasks, when prompted with either a few examples (Ye and Ren, 2021; Bragg et al., 2021) or language deﬁnitions (Efrat and Levy, 2020; Weller et al., 2020; Zhong et al., 2021; Mishra et al., 2022b,a; Parmar et al.,…”
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, methodology): “The concurrent work of Bragg et al. (2021) unifies the evaluation for few-shot learning; their zero-shot evaluation setup is the closest to ours, and they used templates and verbalizers (Schick and Schütze, 2020a) to specify the semantics of a task.”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, background): “Recently, there has been a surge of interest in few-shot learning, where language models learn a specific task from a few examples [46, 3].”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “This setting is attracting attention in NLP community rapidly and is also explored in very recent work (Zhong et al., 2021; Mishra et al., 2021; Bragg et al., 2021; Wei et al., 2021).”
Do Prompt-Based Models Really Understand the Meaning of Their Prompts? (2021, methodology): “The prompts can be manually written (Schick and Schütze, 2021a; Bragg et al., 2021) or automatically generated (Gao et al.”
PPT: Pre-trained Prompt Tuning for Few-shot Learning (2021, background): “…2020) and PET(Schick and Schütze, 2021a) which demonstrates the superiority of PLMs in few-shot scenarios, some later works Perez et al. (2021); Bragg et al. (2021) also discuss reasonable few-shot settings by restricting the size of validation set and proposing a uniﬁed framework to evaluate…”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Moreover, benchmarks like FewGLUE [277], FLEX [276], and FewCLUE [275] which evaluate fewshot learning techniques should be extended to other”
CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation (2022, methodology): “Given that UniﬁedQA-v1 (Khashabi et al., 2020) has been effective for tasks beyond QA (Bragg et al., 2021; Marasovi ´ c et al., 2022 models are strong but overlooked baselines in recent works on large-scale models.”
RAFT: A Real-World Few-Shot Text Classification Benchmark (2021, methodology): “Existing few-shot classification benchmarks are typically designed to highlight areas where models fall short [29] or to study particular model abilities [5, 37, 21].”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, background): “…on unifying task formats to maximize the beneﬁts of human annotations, including question answering (Zhong et al., 2021), textual entailment (Yin et al., 2019, 2020; Wang et al., 2021a), and many other tasks (McCann et al., 2018; Keskar et al., 2019; Raffel et al., 2020; Bragg et al., 2021).”
Benchmarking Generalization via In-Context Instructions on 1, 600+ Language Tasks (2022, background): “A recent literature is motivated by building models that are generalizable across a variety of language tasks, either when prompted with a few examples (Ye and Ren, 2021; Min et al., 2021; Bragg et al., 2021) or by language definitions (Efrat and Levy, 2020; Weller et al.”
PRIMER: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization (2021, methodology): “…datasets we consider, and also the results in the few-shot setting might be inﬂuenced by sampling variability (especially with only 10 examples) (Bragg et al., 2021), we run the same experiments for the compared models ﬁve times with different random seeds (shared with all the models), with the…”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, background): “Going beyond multi-task training, these approaches show better generalization to unseen tasks when prompted with a few examples (Bragg et al., 2021; Min et al., 2022a,b) or language definitions and constraints (Weller et al., 2020; Zhong et al., 2021b; Xu et al., 2022).”
UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training (2022, result): “Followup works have also replicated similar successes on more recent datasets that did not exist at the time of UNIFIEDQA’s construction (Hendrycks et al., 2020; Dasigi et al., 2021; Bragg et al., 2021; Wu et al., 2021; Zhong et al., 2021b) – further strengthening its empirical strength.”
Repository-Level Prompt Generation for Large Language Models of Code (2022, background): “The first category corresponds to producing continuous/soft prompts where the prompt is described in the latent space of a language model (Li & Liang, 2021; Qin & Eisner, 2021; Bragg et al., 2021; Lester et al., 2021; Liu et al., 2021).”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, background): “Finally, many benchmarks have been proposed for comparing few-shot approaches in a standardized way (e.g., Mishra et al., 2021; Bragg et al., 2021; Xu et al., 2021; Ye et al., 2021; Alex et al., 2021).”
FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding (2021, background): “FLEX (Bragg et al., 2021) and CrossFit (Ye et al., 2021) studied principles of designing tasks, datasets, and metrics.”
A Survey on Data Augmentation for Text Classification (2021, unknown): “few-shot learning [156] or natural language generation [157].”