TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency

By Adji Bousso Dieng, Chong Wang, Jianfeng Gao, John William Paisley. 2016

251 citations recorded by the source corpus.

In this paper, we propose TopicRNN, a recurrent neural network (RNN)-based language model designed to directly capture the global semantic meaning relating words in a document via latent topics. Because of their sequential nature, RNNs are good at capturing the local structure of a word sequence – both semantic and syntactic – but might face difficulty remembering long-range dependencies. Intuitively, these long-range dependencies are of semantic nature. In contrast, latent topic models are able to capture the global underlying semantic structure of a document but do not account for word ordering. The proposed TopicRNN model integrates the merits of RNNs and latent topic models: it captures local (syntactic) dependencies using an RNN and global (semantic) dependencies using latent topics. Unlike previous work on contextual RNN language modeling, our model is learned end-to-end. Empirical results on word prediction show that TopicRNN outperforms existing contextual RNN baselines. In addition, TopicRNN can be used as an unsupervised feature extractor for documents. We do this for sentiment analysis and report a new state-of-the-art error rate on the IMDB movie review dataset that amounts to a 13.3% improvement over the previous best result. Finally TopicRNN also yields sensible topics, making it a useful alternative to document models such as latent Dirichlet allocation.

Subfields: Artificial Intelligence, General Social Sciences, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Natural Language Processing

In Model Architectures

Citations in the corpus

Learned in Translation: Contextualized Word Vectors (2017, neutral): “…2016b] 53 [Dai and Le, 2015] 92.8 NTI [Munkhdalai and Yu, 2016b] 87.3 bmLSTM [Radford et al., 2017] 92.9 re-read LSTM [Sha et al., 2016] 87.5 TRNN [Dieng et al., 2016] 93.8 btree-LSTM [Paria et al., 2016] 87.6 oh-LSTM [Johnson and Zhang, 2016] 94.1 600D ESIM [Chen et al., 2016] 88.0 Virtual…”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “, 2015; Wang and Cho, 2015) to others that rely on document-level topics learned from data (Dieng et al., 2016; Wang et al., 2017).”
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, background): “The idea of capturing document-level semantic information has been previously explored for recurrent neural networks (Mikolov and Zweig, 2012; Ghosh et al., 2016; Dieng et al., 2017), however, we are not aware of any existing convolutional models.”
Learning to Generate Reviews and Discovering Sentiment (2017, methodology): “Combining language modelling with topic modelling and fitting a small supervised feature extractor on top has also achieved strong results on in-domain document level sentiment analysis (Dieng et al., 2016).”
Deconvolutional Paragraph Representation Learning (2017, methodology): “Identifying related topics or sentiments, and abstracting (short) summaries from user generated content such as blogs or product reviews, has recently received significant interest [1, 3, 4, 30, 31, 13, 11].”
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders (2019, methodology): “…et al., 2018), deploying less powerful de-coders (Yang et al., 2017; Semeniuta et al., 2017), using lossy input (Chen et al., 2017), utilizing different latent variable connections (Dieng et al., 2017; 2018; Park et al., 2018), or changing the prior (Tomczak & Welling, 2018; Xu & Durrett, 2018).”
A Survey on Text Classification: From Shallow to Deep Learning (2020, methodology): “…Subj, TREC, etc. multi-task [48] Accuracy SST-1, SST-2, Subj, IMDB distant supervision [63] Accuracy SS-Twitter, SE1604, etc. global dependencies [57] Error Rate IMDB virtual adversarial training [55] Error Rate IMDB, DBpedia, RCV1, etc. capsule [119] Accuracy MR, SST-1, Hospital Feedback basic…”
A Survey on Text Classification: From Traditional to Deep Learning (2020, methodology): “2017 TopicRNN [83] ICML SA [84] Error Rate IMDB 2017 Miyato et al.”
Discovering Discrete Latent Topics with Neural Variational Inference (2017, unknown): “…these, such as the Replicated Softmax (Hinton & Salakhutdinov, 2009), the Auto-Regressive Document Model (Larochelle & Lauly, 2012), Sigmoid Belief Document Model (Mnih & Gregor, 2014), Variational Auto-Encoder Document Model (NVDM) (Miao et al., 2016) and TopicRNN Model (Dieng et al., 2016).”
Focus Attention: Promoting Faithfulness and Diversity in Summarization (2021, methodology): “Recently, document-level topic information has been used for improving neural language models (Mikolov and Zweig, 2012; Ghosh et al., 2016; Dieng et al., 2017; Karmaker Santu et al., 2019), neural response generators (Xing et al., 2017; Dziri et al., 2019), and not surprisingly, neural summarizers…”
Learning low dimensional word based linear classifiers using Data Shared Adaptive Bootstrap Aggregated Lasso with application to IMDb data (2018, result): “ts to about top 150 200 features. There are other approaches in ML and ANN literature where one delves into deeper learning of using sentence model in NLP. A summary of such results can be found in ( [9], [27]). There are several other model selection techniques like subset regression, AIC, BIC, etc”
AI in Health: State of the Art, Challenges, and Future Directions (2019, methodology): “[37] leveraged TopicRNN [38], which combines RNN and global topic modeling to predict CHF patient readmission risk using EHR sequences, where each global topic corresponds to a specific distribution of the events in the EHR sequence.”
Transformer-XL: Language Modeling with Longer-Term Dependency (2018, background): “Existing works range from ones where context representations are manually deﬁned (Mikolov & Zweig, 2012; Ji et al., 2015; Wang & Cho, 2015) to others that rely on document-level topics learned from data (Dieng et al., 2016; Wang et al., 2017).”
Sentiment Classification Using Document Embeddings Trained with Cosine Similarity (2019, unknown): “76 (Dieng et al., 2017) One-hot bi-LSTM 94.”
Topic Compositional Neural Language Model (2017, methodology): “Such models may consequently fail to capture long-term dependencies of a document (Dieng et al., 2016).”
Avoiding Latent Variable Collapse With Generative Skip Models (2018, methodology): “vaes have been studied for text analysis (Bowman et al., 2015; Dieng et al., 2016), collaborative filtering (Liang et al.”
Adversarially Regularized Autoencoders (2017, methodology): “…successes through employing convolutional decoders (Yang et al., 2017; Semeniuta et al., 2017), training the latent representation as a topic model (Dieng et al., 2017; Wang et al., 2018), using the von Mises–Fisher distribution (Guu et al., 2017), and combining VAE with iterative inference (Kim…”