Entailment as Few-Shot Learner

By Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, Hao Ma. 2021

198 citations recorded by the source corpus.

Large pre-trained language models (LMs) have demonstrated remarkable ability as few-shot learners. However, their success hinges largely on scaling model parameters to a degree that makes it challenging to train and serve. In this paper, we propose a new approach, named as EFL, that can turn small LMs into better few-shot learners. The key idea of this approach is to reformulate potential NLP task into an entailment one, and then fine-tune the model with as little as 8 examples. We further demonstrate our proposed method can be: (i) naturally combined with an unsupervised contrastive learning-based data augmentation method; (ii) easily extended to multilingual few-shot learning. A systematic evaluation on 18 standard NLP tasks demonstrates that this approach improves the various existing SOTA few-shot learning methods by 12\%, and yields competitive few-shot performance with 500 times larger models, such as GPT-3.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, background): “Other work has shown that good few-shot performance can also be obtained by combining gradient based optimization with textual prompts (Schick and Sch¨utze, 2021b; Gao et al., 2021; Tam et al., 2021; Wang et al., 2021).”
True Few-Shot Learning with Language Models (2021, background): “Several papers claim to circumvent the need to estimate EL ( A , R ) by choosing hyperparameters based on an educated guess [3, 9, 14].”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “We do not focus on these techniques in this work; instead, we unify all tasks and systematically compare them. and Wang et al. (2021a) uniﬁed few-shot learning as textual entailment.”
Primer: Searching for Efficient Transformers for Language Modeling (2021, background): “We focus on decoder-only auto-regressive language modeling (LM), because of its generality and success [18, 7, 19, 20, 21].”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “Researchers have explored unifying the formats of different tasks, in order to better enable knowledge transfer, e.g., DecaNLP (McCann et al., 2018), UFO-Entail (Yin et al., 2020) and EFL (Wang et al., 2021).”
Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (2023, methodology): “Transformer-based paraphrase detection models such as EFL (Wang et al., 2021)”
Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning (2021, methodology): “Model answer: Contradiction In this task, a RoBERTa-large model, trained with a few-shot learning method [Wang et al., 2021a], achieves an accuracy of 93 .”
Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanations (2021, methodology): “Until now, the SOTA models in terms of task performance for all ﬁve tasks were models that do not offer any explain-ability (Wang et al., 2020; 2021; Lan et al., 2020; Xie et al., 2019; Yu et al., 2020).”
A Review on Language Models as Knowledge Bases (2022, methodology): “Others also take advantage of this improved performance and evaluate other discrete prompting approaches, such as through entailment (Wang et al., 2021a) and label token optimization (Zhang et al., 2021).”
ChatGPT: Jack of all trades, master of none (2023, methodology): “2%) 981 2 282918 /66664 3 CoLa S EN Linguistic acceptability No Binary classification CoLA [64] / [65] 3 Yes 1042 1042 0 0 (0%) 1042 2 721 /322 4 ColBERT P EN Humor recognition No Binary classification ColBERT [66] / [66] 2 No 40000 1000 5 93 (9.”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, background): “, 2021), textual entailment (Yin et al., 2019, 2020; Wang et al., 2021a), and many other tasks (McCann et al.”
FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark (2021, methodology): “, 2021) and EFL (Wang et al., 2021)) on our newly proposed benchmark.”
Rationale-Inspired Natural Language Explanations with Commonsense (2021, methodology): “In Table 4, we show results for Table 3: Comparison of SOTA models ([45, 46, 22, 48, 49] in order for each dataset) vs. variants of RE X C for prediction performance in all ﬁve predictive tasks.”
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning (2021, background): “Other work has focused on prompt-based/entailment-based few-shot learning approaches (Brown et al., 2020; Schick and Schütze, 2021; Gao et al., 2021; Tam et al., 2021; Wang et al., 2021).”
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning (2022, methodology): “ClinicalXLNET (Huang et al., 2019), and others (Alrowili and Vijay-Shanker, 2021; Kraljevic et al., 2021; Phan et al., 2021), this paradigm is widely used for creating many task-speciﬁc models (Wang et al., 2021a; Banerjee et al., 2021).”
Black-box Prompt Learning for Pre-trained Language Models (2022, methodology): “Following Wang et al. (2021), for CR, we randomly sample 2,000 instances as the test set.”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, background): “[189] hypothesizes that this phenomenon might be because the entailment task is a true language understanding task; a model that performs entailment well is likely to succeed on similarly-framed tasks.”