Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function

By Devendra Singh Sachan, Manzil Zaheer, Ruslan Salakhutdinov. 2019

143 citations recorded by the source corpus.

In this paper, we study bidirectional LSTM network for the task of text classification using both supervised and semisupervised approaches. Several prior works have suggested that either complex pretraining schemes using unsupervised methods such as language modeling (Dai and Le 2015; Miyato, Dai, and Goodfellow 2016) or complicated models (Johnson and Zhang 2017) are necessary to achieve a high classification accuracy. However, we develop a training strategy that allows even a simple BiLSTM model, when trained with cross-entropy loss, to achieve competitive results compared with more complex approaches. Furthermore, in addition to cross-entropy loss, by using a combination of entropy minimization, adversarial, and virtual adversarial losses for both labeled and unlabeled data, we report state-of-theart results for text classification task on several benchmark datasets. In particular, on the ACL-IMDB sentiment analysis and AG-News topic classification datasets, our method outperforms current approaches by a substantial margin. We also show the generality of the mixed objective function by improving the performance on relation extraction task.1

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Unsupervised Data Augmentation for Consistency Training (2019, methodology): “In the fully supervised settings, the pre-BERT SOTAs include ULMFiT [35] for Yelp-2 and Yelp-5, DPCNN [36] for Amazon-2 and Amazon-5, Mixed VAT [37] for IMDb and DBPedia.”
A review on the long short-term memory model (2020, methodology): “In Sachan et al. (2019) the BLSTM network was explored for text classiﬁcation using both supervised and semi-supervised learning.”
SentiLARE: Linguistic Knowledge Enhanced Language Representation for Sentiment Analysis (2019, methodology): “…We also chose some task-speciﬁc baselines without pre-training for corresponding tasks, including SC-SNN (Chen et al., 2019), DRNN (Wang, 2018), ML (Sachan et al., 2019) for sentence-level sentiment classiﬁcation, DE-CNN (Xu et al., 2018) for aspect term extraction, CDT (Sun et al., 2019a) for…”
Unsupervised Data Augmentation (2019, methodology): “Specifically, following VAT [Miyato et al., 2018], we choose to minimize the KL divergence between the predicted distributions on an unlabeled example and an augmented unlabeled example: min θ JUDA(θ) = E x∈U E x̂∼q(x̂|x) [ DKL ( pθ̃(y | x) ∥∥ pθ(y | x̂)))] , (2) where q(x̂ | x) is a data augmentati”
Deep Learning--based Text Classification (2020, unknown): “[174] study LSTM models for semi-supervised TC.”
A Survey on Deep Semi-Supervised Learning (2021, background): “Text classification AG News [95], BPpedia [96], Yahoo! Answers [97], IMDB [98], Yelp review [95], Snippets [99], Ohsumed [100], TagMyNews [101], MR [102], Elec [103], Rotten Tomatoes [102], RCV1 [104] [105], [106], [107], [108], [109], [110]”
SeqVAT: Virtual Adversarial Training for Semi-Supervised Sequence Labeling (2020, methodology): “In natural language processing, ST has been successfully applied to word sense disambiguation (Yarowsky, 1995) and parsing (McClosky et al., 2006), and EM also has successful application in text classiﬁcation (Sachan et al., 2019).”
ProtoAttend: Attention-Based Prototypical Learning (2019, methodology): “Compared to the state of the art models, our canonical method with simple and small models shows similar accuracy by making slightly fewer predictions – e.g. for MNIST, (Wan et al., 2013) achieves 0.21% error rate, that is obtained by our method refraining from only 0.45% of predictions using ResNet”
Attention-Based Prototypical Learning Towards Interpretable, Confident and Robust Deep Neural Networks (2019, methodology): “For DBpedia, (Sachan & Petuum, 2018) achieves 0.91% error, that is obtained by our framework refraining from 3% of predictions.”