Learning Stochastic Recurrent Networks

By Justin Bayer, Christian Osendorfer. 2014

292 citations recorded by the source corpus.

Leveraging advances in variational inference, we propose to enhance recurrent neural networks with latent variables, resulting in Stochastic Recurrent Networks (STORNs). The model i) can be trained with stochastic gradient methods, ii) allows structured and multi-modal conditionals at each time step, iii) features a reliable estimator of the marginal likelihood and iv) is a generalisation of deterministic recurrent neural networks. We evaluate the method on four polyphonic musical data sets and motion capture data.

Subfields: Artificial Intelligence, Generative Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Model Architectures

Citations in the corpus

Generating Sentences from a Continuous Space (2015, background): “While there has been other work on including continuous latent variables in rnn-style models for modeling speech, handwriting, and music (Bayer and Osendorfer, 2015; Chung et al., 2015), these models include separate latent variables per timestep and are unsuitable for our goal of modeling global fe”
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, background): “Finally, several previous works have tried to introduce latent variables into sequence modeling (Bayer & Osendorfer, 2014; Gregor et al., 2015; Chung et al., 2015; Gan et al., 2015; Frac-caro et al., 2016; Chung et al., 2016).”
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images (2015, neutral): “[23], Bayer and Osendorfer [24].”
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues (2016, methodology): “This process is problematic from a probabilistic perspective, because the model is forced to generate all high-level structure locally on a step-by-step basis (Boulanger-Lewandowski, Bengio, and Vincent 2012; Bayer and Osendorfer 2014; Chung et al. 2015; Denton et al. 2015).”
A Recurrent Latent Variable Model for Sequential Data (2015, methodology): “In fact, STORN puts further restrictions on the dependency structure of the approximate inference model.”
Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data (2016, background): “M L ] 2 0 M ay images, audio or other sensory inputs and (iii) scales to large data due to optimization of parameters based on stochastic gradient descent [2].”
Deep Kalman Filters (2015, background): “Bayer & Osendorfer (2014) create a stochastic variant of Recurrent Neural Networks (RNNs) by making the hidden state of the RNN a function of stochastically sampled latent variables at every time step.”
Sequential Neural Models with Stochastic Layers (2016, methodology): “In this section the SRNN is evaluated on the modeling of speech and polyphonic music data, as they have shown to be difficult to model without a good representation of the uncertainty in the latent states [3, 7, 11, 12, 15].”
Stochastic Video Generation with a Learned Prior (2018, background): “Bayer & Osendorfer (2014) introduce stochastic latent variables into a recurrent network in order to model music and motion capture data.”
Structured Inference Networks for Nonlinear State Space Models (2016, methodology): “The baselines we compare to in Table 3 also have more complex generative models than the DMM. STORN has edges from x t − 1 to z t given by the recurrence update and TSBN has edges from x t − 1 to z t as well as from x t − 1 to x t .”
Disentangled Sequential Autoencoder (2018, methodology): “For sequence modelling, a number of prior publications have extended VAE to video and speech data (Fabius & van Amersfoort, 2014; Bayer & Osendorfer, 2014; Chung et al., 2015).”
Improved Variational Autoencoders for Text Modeling using Dilated Convolutions (2017, methodology): “(Chung et al., 2015; Bayer & Osendorfer, 2014; Fraccaro et al., 2016) embedded variational units in every step of a RNN, which is different from our model in using global latent variables to learn high level features.”
Variational Recurrent Auto-Encoders (2014, methodology): “Combining RNNs with SGVB is partly inspired by the work of Justin Bayer, the ﬁrst results of which were presented as workshop at NIPS 2014 (Bayer & Osendorfer, 2014).”
Learning Awareness Models (2018, background): “…lot of recent interest in sequence-to-sequence modelling (Downey et al., 2017; Venkatraman et al., 2017; Chung et al., 2015; Fraccaro et al., 2016; Bayer & Osendorfer, 2014; Archer et al., 2015; Krishnan et al., 2015), particularly in the context of predicting distributions and in dynamics…”
Deep and Hierarchical Implicit Models (2017, background): “This deﬁnes a stochastic RNN (Bayer & Osendorfer, 2014; Fraccaro et al., 2016) without deterministic connections.”
A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music (2018, background): “Finally, there have been many recurrent models proposed where the recurrent states are themselves stochastic latent variables with dependencies across time (Chung et al., 2015; Bayer & Osendorfer, 2014; Fraccaro et al., 2016).”
Continual Learning in Recurrent Neural Networks with Hypernetworks (2020, unknown): “[58] Justin Bayer and Christian Osendorfer.”
Multiple Futures Prediction (2019, background): “Connections to other Stochastic RNNs Various stochastic recurrent models in existing literature have been proposed: DRAW [20], STORN [3], VRNN [11], SRNN [18], Z-forcing [19], Graph-VRNN [31].”
Deep Temporal Sigmoid Belief Networks for Sequence Modeling (2015, background): “and all the recent recurrent extensions of it [27, 28, 29].”