Generative and Discriminative Text Classification with Recurrent Neural Networks

By Dani Yogatama, Chris Dyer, Wang Ling, Phil Blunsom. 2017

214 citations recorded by the source corpus.

We empirically characterize the performance of discriminative and generative LSTM models for text classification. We find that although RNN-based generative models are more powerful than their bag-of-words ancestors (e.g., they account for conditional dependencies across words in a document), they have higher asymptotic error rates than discriminatively trained RNN models. However we also find that generative models approach their asymptotic error rate more rapidly than their discriminative counterparts---the same pattern that Ng & Jordan (2001) proved holds for linear classification models that make more naive conditional independence assumptions. Building on this finding, we hypothesize that RNN-based generative classification models will be more robust to shifts in the data distribution. This hypothesis is confirmed in a series of experiments in zero-shot and continual learning settings that show that generative models substantially outperform discriminative models.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

GeDi: Generative Discriminator Guided Sequence Generation (2020, background): “This ability likely emerges because generative classifiers can classify unseen topics zeroshot from learned word embeddings (Yogatama et al., 2017), and GeDi uses a generative classifier to guide generation.”
Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach (2019, background): “Yogatama et al. (2017) build generative LSTM to generate text given the embedded labels.”
How to Fine-Tune BERT for Text Classification? (2019, neutral): “We compare our model with the following a variety of different methods: CNN-based methods such as Char-level CNN (Zhang et al., 2015), VD-CNN (Conneau et al., 2016) and DPCNN (Johnson and Zhang, 2017); RNN-based models such as D-LSTM (Yogatama et al., 2017), Skim-LSTM (Seo et al., 2017) and hierarch”
Noisy Channel Language Model Prompting for Few-Shot Text Classification (2021, background): “, 2019) and their extensions to other tasks (Yogatama et al., 2017; Lewis and Fan, 2018).”
Do Convolutional Networks need to be Deep for Text Classification ? (2017, methodology): “The models from the litterature we compare to are: • bag of words: The BOW model is based on the most frequent words from the training data (Zhang et al., 2015) • ngrams: The bag-of-ngrams model exploits the most frequent word n-grams from the training data (Zhang et al., 2015) • ngrams TFIDF: Same ”
Description Based Text Classification with Reinforcement Learning (2020, background): “In the literature of zero-shot text classiﬁcation, knowledge of labels are incorporated in the form of word embed-dings (Yogatama et al., 2017; Rios and Kavuluru, 2018), or class descriptions (Zhang et al., 2019; Srivastava et al., 2018).”
Densely Connected CNN with Multi-scale Feature Attention for Text Classification (2018, methodology): “, 2017]; RNNs including LSTM and its variants Discriminative-LSTM [Yogatama et al., 2017]; CNNs including classical CNN [Kim, 2014], CNN with dynamic pooling [Blunsom et al.”
Residual Recurrent Highway Networks for Learning Deep Sequence Prediction Models (2018, background): “video interpretation [21, 27], language modeling [25, 28–30], speech recognition [31, 32], machine translation [33] and text classification [34, 35], etc.”
Know thy Corpus! Robust Methods for Digital Curation of Web corpora (2020, methodology): “The specific Machine Learning model in this study is based on a bi-directional LSTM classifier (Yogatama et al., 2017) with the attention mechanism (Liu and Lane, 2016).”
Hierarchical CVAE for Fine-Grained Hate Speech Classification (2018, methodology): “Recurrent Neural Networks (RNN) are also commonly used for text classification (Tai et al., 2015; Yogatama et al., 2017).”
Deep learning-based electroencephalography analysis: a systematic review (2019, background): “Models composed of recurrent layers are thus encouraged to make use of the temporal structure of data and have shown high performance in natural language processing (NLP) tasks [230, 210].”
Pivot Based Language Modeling for Improved Neural Domain Adaptation (2018, background): “the strong performance of LSTM and CNN in text classification tasks (Yogatama et al., 2017).”
Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health Record (2018, background): “Many variants of the RNN network have been proposed to address this issue, and long short term memory (LSTM) is one of the most popular models used nowadays in NLP tasks [7], [8], [13]– [16].”
Gradient-based Constrained Sampling from Language Models (2022, background): “• Generative Classifiers Prior work has shown that discriminative classifiers can be fragile to domain shift or adversarial examples (Yogatama et al., 2017; Krause et al., 2020b).”