Neural Variational Inference for Text Processing

By Yishu Miao, Lei Yu, Phil Blunsom. 2015

679 citations recorded by the source corpus.

Recent advances in neural variational inference have spawned a renaissance in deep latent variable models. In this paper we introduce a generic variational inference framework for generative and conditional models of text. While traditional variational methods derive an analytic approximation for the intractable distributions over latent variables, here we construct an inference network conditioned on the discrete text input to provide the variational distribution. We validate this framework on two very different text modelling applications, generative document modelling and supervised question answering. Our neural variational document model combines a continuous stochastic document representation with a bag-of-words generative model and achieves the lowest reported perplexities on two standard test corpora. The neural answer selection model employs a stochastic representation layer within an attention mechanism to extract the semantics between a question and answer pair. On two question answering benchmarks this model exceeds all previous published benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Probabilistic & Bayesian Methods

Citations in the corpus

Generating Sentences from a Continuous Space (2015, methodology): “In a recent paper with goals similar to ours, Miao et al. (2015) introduce an eﬀective VAE-based document-level language model that models texts as bags of words, rather than as sequences.”
Key-Value Memory Networks for Directly Reading Documents (2016, methodology): “…published results on T REC QA and W IKI QA have been obtained by either convolutional neural networks (Santos et al. , 2016; Yin and Schütze, 2015; Wang et al. , 2016) or recurrent neural networks (Miao et al. , 2015) – both usually with attention mechanisms inspired by (Bahdanau et al. , 2015).”
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents (2019, methodology): “Following the work of Radford et al., the model is pre-trained on the BooksCorpus dataset Zhu et al. (2015) which contains over 7,000 unpublished books (about 800M words) from a variety of genres (Adventure, Fantasy, Romance.”
Variational Autoencoders for Collaborative Filtering (2018, background): “Miao et al. [31] study vaes on text data.”
On the challenges of learning with inference networks on sparse, high-dimensional data (2017, background): “We follow the preprocessing procedure in Miao et al. (2016).”
Incremental Learning from Scratch for Task-Oriented Dialogue Systems (2019, background): “Fortunately, we can obtain its lower bound (Hoffman et al., 2013; Miao et al., 2016; Sohn et al., 2015) as follows:”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “…models for real datasets (e.g., text, images, etc.) already work with a relatively simple and disentangled prior p ( z ) having no interaction among the latent dimensions (e.g., the standard Gaussian N (0 , I )) (Bowman et al., 2015; Miao et al., 2016; Hou et al., 2017; Zhao et al., 2017).”
Neural Semantic Encoders (2016, neutral): “6747 Three-layer LSTMs attention [26] 0.”
Maximum-Likelihood Augmented Discrete Generative Adversarial Networks (2017, background): “…is that it is in-applicable to those auto-regressive models with latent random variables, which have performed better than autoregressive (deterministic state) recurrent neural networks (i.e. usual RNNs, LSTMs or GRUs) on multiple tasks (Serban et al., 2017; Miao et al., 2016; Zhang et al., 2016a).”
Neural Tree Indexers for Text Understanding (2016, methodology): “We conducted experiments on the Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015a (Miao et al., 2016) 0.6552 0.6747 Three-layer LSTM attention (Miao et al., 2016) 0.6639 0.6828 NASM (Miao et al., 2016) 0.6705 0.6914 NTI (Ours) 0.6742 0.6884 batch size to 32.”
Structured Disentangled Representations (2018, methodology): “We train a standard NVDM with a 50-dimensional latent variable using the normal VAE objective.”
Improved Variational Autoencoders for Text Modeling using Dilated Convolutions (2017, methodology): “Related work (Miao et al., 2016; Larochelle & Lauly, 2012; Mnih & Gregor, 2014) has used simpler decoders that model text as a bag of words.”
Deconvolutional Latent-Variable Model for Text Sequence Matching (2017, methodology): “(Miao, Yu, and Blunsom 2016) applied the NVI framework to an unsupervised bags-of-words model.”
TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency (2016, methodology): “Following the proposed variational autoencoder technique, we choose the form of q ( θ ) to be an inference network using a feed-forward neural network (Kingma and Welling, 2013; Miao et al., 2015).”
A Hybrid Convolutional Variational Autoencoder for Text Generation (2017, methodology): “, 2016), their application to natural text generation requires additional care (Bowman et al., 2016; Miao et al., 2015).”
Spherical Latent Spaces for Stable Variational Autoencoders (2018, methodology): “Speciﬁcally, the Neural Variational Document Model (NVDM), proposed by Miao et al. (2016), is a VAE-based unsupervised document model.”
Explaining and Generalizing Back-Translation through Wake-Sleep (2018, methodology): “Inference networks have been applied to a wide variety of problems, e.g., topic modeling (Miao et al., 2016) and inﬂec-tion generation (Zhou and Neubig, 2017).”