Sequential Neural Models with Stochastic Layers

By Marco Fraccaro, Søren Kaae Sønderby, Ulrich Paquet, Ole Winther. 2016

441 citations recorded by the source corpus.

How can we efficiently propagate uncertainty in a latent state representation with recurrent neural networks? This paper introduces stochastic recurrent neural networks which glue a deterministic recurrent neural network and a state space model together to form a stochastic and sequential neural generative model. The clear separation of deterministic and stochastic layers allows a structured variational inference network to track the factorization of the model's posterior distribution. By retaining both the nonlinear recursive structure of a recurrent neural network and averaging over the uncertainty in a latent path, like a state space model, we improve the state of the art results on the Blizzard and TIMIT speech modeling data sets by a large margin, while achieving comparable performances to competing methods on polyphonic music modeling.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Model Architectures

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, background): “Finally, several previous works have tried to introduce latent variables into sequence modeling (Bayer & Osendorfer, 2014; Gregor et al., 2015; Chung et al., 2015; Gan et al., 2015; Fraccaro et al., 2016; Chung et al., 2016).”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “…models with latent variables following the inﬂuential work by Kingma and Welling [37], Rezende et al. [59], and models optimized with this bound have been successfully used to model data such as natural images [58, 38, 11, 26], speech and music time-series [12, 23, 39], and video [2, 28, 17].”
Learning and Querying Fast Generative Models for Reinforcement Learning (2018, background): “…models We build directly on a plethora of recent work exploring the continuum of models ranging from standard recurrent neural networks (RNNs) to fully stochastic models with uncertainty (Chung et al., 2015; Archer et al., 2015; Fraccaro et al., 2016; Krishnan et al., 2015; Gu et al., 2015).”
Variational Lossy Autoencoder (2016, positive): “…of models have run into the problem that the autoregressive part of the model ends up explaining all structure in the data, while the latent variables are not used (Fabius & van Amersfoort, 2014; Chung et al., 2015; Bowman et al., 2015; Serban et al., 2016; Fraccaro et al., 2016; Xu & Sun, 2016).”
Deep Factors for Forecasting (2019, background): “…(Krishnan et al., 2017; 2015), keep the Gaussian transition dynamics with mean and co-variance matrix parameterized by MLPs. Stochastic RNNs (SRNNs) (Fraccaro et al., 2016) explicitly incorporate the deterministic dynamics from RNNs that do not depend on latent variables by interlacing them with a…”
RAVE: A variational autoencoder for fast and high-quality neural audio synthesis (2021, background): “Recent works in deep audio modelling have allowed novel types of interaction such as unconditional generation (Chung et al., 2015; Fraccaro et al., 2016; Oord et al., 2016; Vasquez & Lewis, 2019; Dhariwal et al., 2020) or timbre transfer between instruments (Mor et al.”
A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning (2017, methodology): “In particular, deep neural networks can parameterize the transition and emission distributions of different variants of deep state-space models [8, 16, 19].”
Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data (2017, background): “While there has been much research investigating static data, such as the aforementioned ones, there is relatively little research on learning from sequential data [8, 3, 2, 9, 7, 18, 36].”
Stochastic Video Generation with a Learned Prior (2018, background): “Several additional works train stochastic recurrent neural networks to model speech, handwriting, natural language (Chung et al., 2015; Fraccaro et al., 2016; Bowman et al., 2016), perform counterfactual inference (Krishnan et al.”
Disentangled Sequential Autoencoder (2018, methodology): “We follow Fraccaro et al. (2017) to simulate video sequences of a ball (or a square) bouncing inside an irregular polygon using Pymunk.”
Human-level performance in 3D multiplayer games with population-based reinforcement learning (2018, background): “Whereas some previous hierarchical RL agents construct explicit hierarchical goals or skills (29–32), this agent architecture is conceptually more closely related to work outside of RL on building hierarchical temporal representations (33–36) and recurrent latent variable models for sequential data ”
Learning Awareness Models (2018, background): “…There has been a lot of recent interest in sequence-to-sequence modelling (Downey et al., 2017; Venkatraman et al., 2017; Chung et al., 2015; Fraccaro et al., 2016; Bayer & Osendorfer, 2014; Archer et al., 2015; Krishnan et al., 2015), particularly in the context of predicting distributions…”
A Hybrid Convolutional Variational Autoencoder for Text Generation (2017, background): “, 2016) and speech (Fraccaro et al., 2016), their application to natural text generation requires additional care (Bowman et al.”
Deep and Hierarchical Implicit Models (2017, background): “This deﬁnes a stochastic RNN (Bayer & Osendorfer, 2014; Fraccaro et al., 2016) without deterministic connections.”
A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music (2018, background): “Finally, there have been many recurrent models proposed where the recurrent states are themselves stochastic latent variables with dependencies across time (Chung et al., 2015; Bayer & Osendorfer, 2014; Fraccaro et al., 2016).”
PixelVAE++: Improved PixelVAE with Discrete Prior (2019, background): “Early models with autoregressive decoder found that the autoregressive part learns all the features of the data and the latent variables are not used [33, 34, 35, 36, 37, 38].”