Joint Embedding of Words and Labels for Text Classification

By Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ricardo Henao, Lawrence Carin. 2018

428 citations recorded by the source corpus.

Word embeddings are effective intermediate representations for capturing semantic regularities between words, when learning the representations of text sequences. We propose to view text classification as a label-word joint embedding problem: each label is embedded in the same space with the word vectors. We introduce an attention framework that measures the compatibility of embeddings between text sequences and labels. The attention is learned on a training set of labeled samples to ensure that, given a text sequence, the relevant words are weighted higher than the irrelevant ones. Our method maintains the interpretability of word embeddings, and enjoys a built-in ability to leverage alternative sources of information, in addition to input text sequences. Extensive results on the several large text datasets show that the proposed framework outperforms the state-of-the-art methods by a large margin, in terms of both accuracy and speed.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Graph Convolutional Networks for Text Classification (2018, background): “Several recent studies showed that the success of deep learning on text clas-siﬁcation largely depends on the effectiveness of the word embeddings (Shen et al. 2018; Joulin et al. 2017; Wang et al. 2018).”
Deep Learning--based Text Classification (2020, background): “Wang et al. [84] view TC as a label-word matching problem: each label is embedded in the same space with the word vector.”
Hierarchical Contextualized Representation for Named Entity Recognition (2019, background): “Considering that words may contribute differently to the sentence-level representation, we adopt label embedding attention (Wang et al. 2018) to get an attention score for the entire sentence and then transform the hidden states v ∈ R N × d s into a ﬁxed-sized sentence-level representation s ∈ R d…”
Learning Context-Sensitive Convolutional Filters for Text Processing (2017, background): “…various natural language processing applications (Collobert et al., 2011), including sentence/document classiﬁcation (Kim, 2014; Zhang et al., 2015; Wang et al., 2018), text sequence matching (Hu et al., 2014; Yin et al., 2016; Shen et al., 2017), generic text representations (Gan et al., 2016;…”
Towards Better Representations with Deep/Bayesian Learning (2018, background): “For example, deep generative models for natural language understanding and generation (Gan et al., 2017d; Zhang et al., 2017a,b; Shen et al., 2017, 2018; Wang et al., 2018).”
BertGCN: Transductive Text Classification by Combining GNN and BERT (2021, background): “Other works leverage label embeddings and jointly train them along with input texts (Wang et al., 2018; Pappas and Henderson, 2019).”
A Survey on Text Classification: From Shallow to Deep Learning (2020, neutral): “[55] ICLR SA 246 [56] 2017 TopicRNN [57] ICML SA 113 [58] 2017 DPCNN [59] ACL SA, TL 156 [60] 2017 IAN [61] IJCAI SA 222 [62] 2017 DeepMoji [63] EMNLP SA 260 [64] 2017 RAM [65] EMNLP SA 225 [62] 2018 ELMo [66] NAACL SA, QA, NLI 3722 [67] 2018 DGCNN [68] TheWebConf TL 81 [69] 2018 ULMFiT [70] ACL SA,”
Few-Shot Named Entity Recognition: A Comprehensive Study (2020, methodology): “…(Vinyals et al., 2016; Sung et al., 2018; Zhao et al., 2020), and has recently been adapted to different NLP tasks such as text classiﬁcation (Wang et al., 2018; Geng et al., 2019; Bansal et al., 2020), machine translation (Gu et al., 2018) and relation classiﬁcation (Han et al., 2018).”
Federated Learning for ICD Classification with Lightweight Models and Pretrained Embeddings (2025, unknown): “The research focus was then shifted to more sophisticated techniques, like attention mechanisms (e.g., CAML, DR-CAML) and label-aware embeddings (e.g., LEAM), which improved the processing of multi-label outputs and rare codes [4,7].”