Variational Recurrent Auto-Encoders

By Otto Fabius, Joost R. van Amersfoort, Diederik P. Kingma. 2014

269 citations recorded by the source corpus.

In this paper we propose a model that combines the strengths of RNNs and SGVB: the Variational Recurrent Auto-Encoder (VRAE). Such a model can be used for efficient, large scale unsupervised learning on time series data, mapping the time series data to a latent vector representation. The model is generative, such that data can be generated from samples of the latent space. An important contribution of this work is that the model can make use of unlabeled data in order to facilitate supervised training of RNNs by initialising the weights and network state.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generating Sentences from a Continuous Space (2015, methodology): “…has been some work on discrete sequences: a technique for doing this using rnn encoders and de-coders, which shares the same high-level architecture as our model, was proposed under the name Variational Recurrent Autoencoder ( vrae ) for the modeling of music in Fabius and van Amersfoort (2014).”
Variational Lossy Autoencoder (2016, neutral): “…of models have run into the problem that the autoregressive part of the model ends up explaining all structure in the data, while the latent variables are not used (Fabius & van Amersfoort, 2014; Chung et al., 2015; Bowman et al., 2015; Serban et al., 2016; Fraccaro et al., 2016; Xu & Sun, 2016).”
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues (2016, neutral): “Similar to our work is the Variational Recurrent Autoencoder [7] and the Variational Autoencoder Language Model [3], which apply encoder-decoder architectures to generative music modeling and language modeling respectively.”
A Recurrent Latent Variable Model for Sequential Data (2015, background): “[4], Bayer and Osendorfer [2], Fabius and van Amersfoort [6], we believe we are the first to integrate these dependencies between the latent random variables at neighboring timesteps.”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, methodology): “Other generative models based on similar principles are Variational Recurrent AEs [85], PixelGAN AEs [86] and Adversarial Symmetric Variational AEs [87].”
Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data (2017, background): “While there has been much research investigating static data, such as the aforementioned ones, there is relatively little research on learning from sequential data [8, 3, 2, 9, 7, 18, 36].”
Sequential Neural Models with Stochastic Layers (2016, methodology): “A number of works have extended RNNs with stochastic units to model motion capture, speech and music data [3, 7, 11, 12, 15].”
Disentangled Sequential Autoencoder (2018, methodology): “For sequence modelling, a number of prior publications have extended VAE to video and speech data (Fabius & van Amersfoort, 2014; Bayer & Osendorfer, 2014; Chung et al., 2015).”
Generating Diverse and Natural Text-to-Speech Samples Using a Quantized Fine-Grained VAE and Autoregressive Prosody Prior (2020, methodology): “Attempts to introduce temporal correlation to sequential latent representations [15,16] often adopt autoregressive decomposition of the prior and posterior distributions, parameterizing both using neural networks.”
A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music (2018, methodology): “A very similar model was also proposed by (Fabius & van Amersfoort, 2015), which was applied with limited success to music.”
PixelVAE++: Improved PixelVAE with Discrete Prior (2019, background): “Early models with autoregressive decoder found that the autoregressive part learns all the features of the data and the latent variables are not used [33, 34, 35, 36, 37, 38].”
Sync-DRAW: Automatic Video Generation using Deep Recurrent Attentive Architectures (2016, background): “…to generate the video sequence by sampling from the latent distribution learnt by the VAE at each time step. core components of a standard R-VAE are the encoder LSTM, LSTM enc (which outputs h enc ), the latent representation, Z , and the decoder LSTM, LSTM dec (which outputs h dec ) [2].”
Data Augmentation for Spoken Language Understanding via Joint Variational Generation (2018, background): “Notably, variational recurrent auto-encoders (VRAE) were ﬁrst proposed by (Fabius and van Amersfoort 2014).”
Deep Learning Techniques for Music Generation - A Survey (2017, background): “In [43], Fabius and van Amersfoort propose the extension of the RNN Encoder-Decoder architecture to the case of a variational autoencoder (VAE), which is therefore named a variational recurrent autoencoder (VRAE).”
Generating Multi-Agent Trajectories using Programmatic Weak Supervision (2018, methodology): “We use a VRAE-style model from (Fabius & van Amersfoort, 2014) in which we encode an entire trajectory into a latent macro-intent variable z , with the idea that z should encode global properties of the sequence.”