Few-shot Text Classification with Distributional Signatures

By Yujia Bao, Menghua Wu, Shiyu Chang, Regina Barzilay. 2019

184 citations recorded by the source corpus.

In this paper, we explore meta-learning for few-shot text classification. Meta-learning has shown strong performance in computer vision, where low-level patterns are transferable across learning tasks. However, directly applying this approach to text is challenging--lexical features highly informative for one task may be insignificant for another. Thus, rather than learning solely from words, our model also leverages their distributional signatures, which encode pertinent word occurrence patterns. Our model is trained within a meta-learning framework to map these signatures into attention scores, which are then used to weight the lexical representations of words. We demonstrate that our model consistently outperforms prototypical networks learned on lexical knowledge (Snell et al., 2017) in both few-shot text classification and relation classification by a significant margin across six benchmark datasets (20.0% on average in 1-shot classification).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Making Pre-trained Language Models Better Few-shot Learners (2021, background): “…2020; Chen et al., 2020), where a set of unlabeled examples are given; (2) meta-learning (Yu et al., 2018; Han et al., 2018; Bansal et al., 2020a,b; Bao et al., 2020), where a set of auxiliary tasks are given; and (3) intermediate training (Phang et al., 2018; Yin et al., 2020), where a related,…”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “More recently, attention was also used to summarize the distribution of an incoming text classiﬁcation task [129].”
FLEX: Unifying Evaluation for Few-Shot NLP (2021, methodology): “This suggests that the choices of |Etest| = 1, 5, 10 in prior work [4, 24, 56, 75] can mean inaccurate and wide CIs, while choices of |Etest| = 1000 [5] can be prohibitively costly for methods with high training cost.”
Few-shot Conformal Prediction with Auxiliary Tasks (2021, background): “…a popular approach to transferring knowledge gained from auxiliary tasks—e.g., via featurizations or statistics (Edwards & Storkey, 2017)— to a target task that is otherwise resource-limited (Vinyals et al., 2016; Finn et al., 2017; Snell et al., 2017; Bertinetto et al., 2019; Bao et al., 2020).”
DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference (2021, background): “, 2020), domains (Bao et al., 2020; Yu et al., 2018; Geng et al., 2019), and language pairs (Gu et al.”
A Survey on Text Classification: From Shallow to Deep Learning (2020, background): “…character encoding [122] Accuracy Geonames toponyms, etc. dynamic routing [78] Accuracy Subj, TREC, Reuters, etc. hierarchical relations [123] M icro − F 1, M acro − F 1, etc. RCV1, Amazon670K meta-learning [124] Accuracy 20NG, RCV, Reuters-2157, etc. hierarchical attention [46] Accuracy Yelp.”
Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks (2019, background): “, 2016) model in a multi-task setting across various sentiment classification domains, and other recent approaches (Guo et al., 2018; Yu et al., 2018; Han et al., 2018; Obamuyide and Vlachos, 2019; Geng et al., 2019; Mi et al., 2019; Bao et al., 2020) meta-train for a specific classification task, s”
Few-shot Sequence Learning with Transformers (2020, background): “In the sequential data setting, Bao et al. (2019) constructed synthetic benchmarks from existing text datasets but the number of tasks is rather limited.”
Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners (2021, methodology): “The pre-train—ﬁne-tune paradigm has become the de facto standard for natural language processing (NLP), and has achieved excellent results in several benchmarks (Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020; Dong et al., 2019; Bao et al., 2020a).”