Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach

By Wenpeng Yin, Jamaal Hay, Dan Roth. 2019

684 citations recorded by the source corpus.

Zero-shot text classification (0Shot-TC) is a challenging NLU problem to which little attention has been paid by the research community. 0Shot-TC aims to associate an appropriate label with a piece of text, irrespective of the text domain and the aspect (e.g., topic, emotion, event, etc.) described by the label. And there are only a few articles studying 0Shot-TC, all focusing only on topical categorization which, we argue, is just the tip of the iceberg in 0Shot-TC. In addition, the chaotic experiments in literature make no uniform comparison, which blurs the progress. This work benchmarks the 0Shot-TC problem by providing unified datasets, standardized evaluations, and state-of-the-art baselines. Our contributions include: i) The datasets we provide facilitate studying 0Shot-TC relative to conceptually different and diverse aspects: the “topic” aspect includes “sports” and “politics” as labels; the “emotion” aspect includes “joy” and “anger”; the “situation” aspect includes “medical assistance” and “water shortage”. ii) We extend the existing evaluation setup (label-partially-unseen) – given a dataset, train on some labels, test on all labels – to include a more challenging yet realistic evaluation label-fully-unseen 0Shot-TC (Chang et al., 2008), aiming at classifying text snippets without seeing task specific training data at all. iii) We unify the 0Shot-TC of diverse aspects within a textual entailment formulation and study it this way.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “Other approaches for few-shot learning in NLP include exploiting examples from related tasks (Yu et al., 2018; Gu et al., 2018; Dou et al., 2019; Qian and Yu, 2019; Yin et al., 2019) and using data augmentation (Xie et al.”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “Our work also falls in the well-established category of zero-shot learning, which has historically been used to refer to classifying instances among a set of unseen categories (Lampert et al., 2009; Romera-Paredes & Torr, 2015; Srivastava et al., 2018; Yin et al., 2019, inter alia).”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, neutral): “, 2022), entailment pairs (Yin et al., 2019; 2020) or question-answer formats (Puri & Catanzaro, 2019; Zhong et al.”
Incremental Few-shot Text Classification with Multi-round New Classes: Formulation, Dataset and System (2021, neutral): “Yin et al. (2019) is another work that utilizes textual entailment for zero-shot text classiﬁcation.”
Learning from Task Descriptions (2020, methodology): “1Data, evaluation code, baseline models, and leaderboard at https://allenai.org/data/zest 2019) and have shown promising results in zero shot prediction by leveraging their language understanding capabilities (Levy et al., 2017; Zhou et al., 2018; Yin et al., 2019).”
Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start (2020, background): “There are more NLP tasks that can ﬁt the entailment framework easily, such as text classiﬁcation (Yin et al., 2019), relation extraction, summarization, etc.”
Zero-shot Text Classification via Reinforced Self-training (2020, methodology): “The ﬁrst comes from the recently released benchmarks for zero-shot text classiﬁcation (Yin et al., 2019), including 3 datasets: topic, emotion and situation classiﬁcation.”
Meta-tuning Language Models to Answer Prompts Better (2021, methodology): “For example, we consider AG News topic classiﬁcation dataset (Zhang et al., 2015) and the topic classiﬁcation dataset from Yin et al. (2019) to be similar, even though their sources and label spaces are different.”
Top2Vec: Distributed Representations of Topics (2020, background): “The Yahoo Answers dataset [36, 37], contains 1.3 million labelled posts.”
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification (2021, background): “However, since the extra classiﬁer requires adequate training instances to tune, it is still challenging to apply ﬁne-tuning in few-shot learning (Brown et al., 2020) and zero-shot learning (Yin et al., 2019) scenarios.”
RAFT: A Real-World Few-Shot Text Classification Benchmark (2021, background): “[38] and Davison [9] as an effective zero-shot classification approach”
Text Generation with Efficient (Soft) Q-Learning (2021, background): “Prominent examples include learning to generate prompts to control the massive LMs [65, 54, 67], learning text generation from noisy or even negative data, learning to generate adversarial text attacks for robustness study [60, 1], and others, where people have to devise specialized algorithms due…”
Learning to Generate Task-Specific Adapters from Task Description (2021, background): “Zeroshot learning (ZSL) has been explored for various NLP tasks, including text classification (Yin et al., 2019), entity linking (Logeswaran et al., 2019) and entity typing (Obeidat et al., 2019).”
Continual Learning with Transformers for Image Classification (2022, background): “Besides, some recent studies [2, 18, 35] showed transformers have a good ability to generalize well”
PERSONACHATGEN: Generating Personalized Dialogues using GPT-3 (2022, methodology): “To verify that GPT-3 generates a profile sentences that are relevant to the given persona category , we leveraged an NLI-based zero-shot classification task (Yin et al., 2019), that classifies a sentence through the "entailment" label predicted by NLI model.”