Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings

By Rie Johnson, Tong Zhang. 2016

263 citations recorded by the source corpus.

One-hot CNN (convolutional neural network) has been shown to be effective for text categorization (Johnson & Zhang, 2015). We view it as a special case of a general framework which jointly trains a linear model with a non-linear feature generator consisting of `text region embedding + pooling'. Under this framework, we explore a more sophisticated region embedding method using Long Short-Term Memory (LSTM). LSTM can embed text regions of variable (and possibly large) sizes, whereas the region size needs to be fixed in a CNN. We seek effective and efficient use of LSTM for this purpose in the supervised and semi-supervised settings. The best results were obtained by combining region embeddings in the form of LSTM and convolution layers trained on unlabeled data. The results indicate that on this task, embeddings of text regions, which can convey complex concepts, are more useful than embeddings of single words in isolation. We report performances exceeding the previous best results on four benchmark datasets.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, methodology): “This is promising as the existing state-of-the-art requires complex architectures (Peters et al., 2018), multiple forms of attention (McCann et al., 2017) and sophisticated embedding schemes (Johnson and Zhang, 2016), while our method employs a regular LSTM with dropout.”
Learned in Translation: Contextualized Word Vectors (2017, neutral): “…87.3 bmLSTM [Radford et al., 2017] 92.9 re-read LSTM [Sha et al., 2016] 87.5 TRNN [Dieng et al., 2016] 93.8 btree-LSTM [Paria et al., 2016] 87.6 oh-LSTM [Johnson and Zhang, 2016] 94.1 600D ESIM [Chen et al., 2016] 88.0 Virtual [Miyato et al., 2017] 94.1 BCN+Char+CoVe (Ours) 88.1 Test Performance.”
Fine-tuned Language Models for Text Classification (2018, methodology): “and TREC-6 datasets, we compare against CoVe (McCann et al., 2017), a stateModel Test Model Test IMDb BCN+Char+CoVe (McCann et al., 2017) 91.8 TREC-6 BCN+Char+CoVe (McCann et al., 2017) 95.8 oh-LSTM (Johnson and Zhang, 2016) 94.1 TBCNN (Mou et al., 2015) 96.0 Virtual (Miyato et al., 2016) 94.1 LSTM-”
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, methodology): “Many complex variations of these models have been proposed to improve the text clas-siﬁcation accuracy, e.g. training one-hot CNN (JZ15a; Johnson and Zhang 2015a) or one-hot bidirectional LSTM (BiL-STM) network with dynamic max-pooling (JZ16; Johnson and Zhang 2016).”
Adversarial Training Methods for Semi-Supervised Text Classification (2016, methodology): “These models now achieve state of the art Bag-of-words(Zhang et al., 2015) 3.57% Large-CNN(character-level) (Zhang et al., 2015) 1.73% SA-LSTM(word-level)(Dai & Le, 2015) 1.41% N-grams TFIDF (Zhang et al., 2015) 1.31% SA-LSTM(character-level)(Dai & Le, 2015) 1.19% Word CNN (Johnson & Zhang, 2016a) 0”
Diverse Few-Shot Text Classification with Multiple Metrics (2018, methodology): “To adapt the MNet to text classification, we choose encoder f to be a convolutional neural network (CNN) following (Kim, 2014; Johnson and Zhang, 2016).”
Deconvolutional Paragraph Representation Learning (2017, methodology): “Instead of pre-training a fully unsupervised model as in [1, 3], we cast the semi-supervised task as a multi-task learning problem similar to [35], i.”
Deep Learning in Mobile and Wireless Networking: A Survey (2018, background): “, speech recognition [175], text categorization [176], and wearable activity recognition [98]).”
Contextual Explanation Networks (2017, methodology): “For IMDB, following Johnson et al. [49] and we use a bi-directional LSTM with max pooling.”
Convolutional Neural Networks for Text Categorization: Shallow Word-level vs. Deep Character-level (2016, background): “However, CNNs are particularly attractive since, due to their simplicity and parallel processing-friendly nature, training and testing of CNNs can be made much faster than LSTM to achieve similar accuracy [5], and therefore CNNs have a potential to scale better to large training data.”
A Survey on Text Classification: From Traditional to Deep Learning (2020, unknown): “The SICK contains almost 10,000 English sentence pairs.”
Deep Learning--based Text Classification (2020, methodology): “Johnson and Rie [39] explore a text region embedding method using LSTM.”
A Survey on Text Classification: From Shallow to Deep Learning (2020, unknown): “…layer [39] Accuracy, M acro − F 1 20NG, Fudan, ACL, SST-2 multi-timescale [8] Accuracy SST-1, SST-2, QC, IMDB RNN embeddings of text regions [117] Error Rate IMDB, Elec, RCV1, 20NG 2DCNN [118] Accuracy SST-1, SST-2, Subj, TREC, etc. multi-task [48] Accuracy SST-1, SST-2, Subj, IMDB distant…”
Investigating the Working of Text Classifiers (2018, background): “The next generation of approaches includes neural networks that have shown to outperform bag-ofwords models in text classification tasks (Kim, 2014; Johnson and Zhang, 2016).”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, unknown): “CNN [103] 91.33 Deep Pyramid CNN [104] 93.13 CNN [105] 93.43 Universal Language Model Fine-tuning (ULMFiT) [106] 94.99”
Bridging the domain gap in cross-lingual document classification (2019, background): “The study of weakly supervised learning in language processing is relatively new (Johnson & Zhang, 2016; Yu et al., 2018).”