A Recurrent Latent Variable Model for Sequential Data

By Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C. Courville, Yoshua Bengio. 2015

1,375 citations recorded by the source corpus.

In this paper, we explore the inclusion of latent random variables into the hidden state of a recurrent neural network (RNN) by combining the elements of the variational autoencoder. We argue that through the use of high-level latent random variables, the variational RNN (VRNN)1 can model the kind of variability observed in highly structured sequential data such as natural speech. We empirically evaluate the proposed model against other related sequential models on four speech datasets and one handwriting dataset. Our results show the important roles that latent random variables can play in the RNN dynamics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, background): “Finally, several previous works have tried to introduce latent variables into sequence modeling (Bayer & Osendorfer, 2014; Gregor et al., 2015; Chung et al., 2015; Gan et al., 2015; Fraccaro et al., 2016; Chung et al., 2016).”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “…models with latent variables following the inﬂuential work by Kingma and Welling [37], Rezende et al. [59], and models optimized with this bound have been successfully used to model data such as natural images [58, 38, 11, 26], speech and music time-series [12, 23, 39], and video [2, 28, 17].”
Density estimation using Real NVP (2016, background): “Its success in leveraging recent advances in backpropagation [51, 39] in deep neural networks resulted in its adoption for several applications ranging from speech synthesis [12] to language modeling [8].”
Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, background): “Other approaches have examined RNNs with richer expressivity, latent-variables for notes, or raw audio synthesis (Boulanger-Lewandowski et al., 2012; Gu et al., 2015; Chung et al., 2015).”
The challenge of realistic music generation: modelling raw audio at scale (2018, methodology): “Recent successful attempts rely on autoregressive (AR) models: WaveNet [43], VRNN [10], WaveRNN [23] and SampleRNN [31] predict digital waveforms one timestep at a time.”
Learning Latent Dynamics for Planning from Pixels (2018, methodology): “We design a latent dynamics model with both deterministic and stochastic components (Buesing et al., 2018; Chung et al., 2015).”
Learning and Querying Fast Generative Models for Reinforcement Learning (2018, background): “…models We build directly on a plethora of recent work exploring the continuum of models ranging from standard recurrent neural networks (RNNs) to fully stochastic models with uncertainty (Chung et al., 2015; Archer et al., 2015; Fraccaro et al., 2016; Krishnan et al., 2015; Gu et al., 2015).”
Variational Lossy Autoencoder (2016, neutral): “The combination of latent variables with expressive decoder was previously explored using recurrent networks mainly in the context of language modeling (Chung et al., 2015; Bowman et al., 2015; Serban et al., 2016; Fraccaro et al., 2016; Xu & Sun, 2016).”
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues (2016, methodology): “This process is problematic from a probabilistic perspective, because the model is forced to generate all high-level structure locally on a step-by-step basis (Boulanger-Lewandowski, Bengio, and Vincent 2012; Bayer and Osendorfer 2014; Chung et al. 2015; Denton et al. 2015).”
Hierarchical Multiscale Recurrent Neural Networks (2016, background): “The most popular approach is to set the timescales as hyperparameters (El Hihi & Bengio, 1995; Koutník et al., 2014; Bahdanau et al., 2016) instead of treating them as dynamic variables that can be learned from the data (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016).”
Variational Continual Learning (2017, background): “By passing a simple noise variable (e.g. Gaussian noise) through a deep neural network, these models have been shown to be able to generate realistic images, sounds and videos sequences (Chung et al., 2015; Kingma et al., 2016; Vondrick et al., 2016).”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, positive): “In particular, given a set of trajectories, we optimize a fully differentiable recurrent variational autoencoder (Chung et al., 2015; Gregor et al., 2015; Kingma & Welling, 2013) with binary latent variables (Nalisnick & Smyth, 2016), each designated to a state whose prior distribution (conditioning”
Causal Effect Inference with Deep Latent-Variable Models (2017, neutral): “Since their recent introduction, VAEs have been shown to be remarkably successful in capturing latent structure across a wide-range of previously difficult problems, such as modeling images [19], volumes [24], time-series [10] and fairness [34].”
Composing graphical models with neural networks for structured representations and fast inference (2016, methodology): “[22] extend the variational autoencoder framework to sequential models, though they focus on RNNs rather than probabilistic graphical models.”
Deep Kalman Filters (2015, methodology): “Chung et al. (2015) model sequences of length T using T variational autoencoders.”
Recurrent Independent Mechanisms (2019, methodology): “Aside from being RIM-specific, the internal operation of the LSTM and GRU remain unchanged, and the active RIMs are updated by h̃t+1,k = Dk(ht,k) = LSTM(ht,k, A (in) k ; θ (D) k ) ∀k ∈ St (2) as a function of the attention mechanism A(in)k applied on the current input, described in the next two subs”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, methodology): “1 VRNNS We start by reviewing the VRNN approach of Chung et al. (2015). This model has three types of variable: the observed output xt, the stochastic VAE state zt, and the deterministic RNN hidden state ht, which summarizes x≤t and z≤t.”