Disentangled Sequential Autoencoder

By Yingzhen Li, Stephan Mandt. 2018

307 citations recorded by the source corpus.

We present a VAE architecture for encoding and generating high dimensional sequential data, such as video or audio. Our deep generative model learns a latent representation of the data which is split into a static and dynamic part, allowing us to approximately disentangle latent time-dependent features (dynamics) from features which are preserved over time (content). This architecture gives us partial control over generating content and dynamics by conditioning on either one of these sets of features. In our experiments on artificially generated cartoon video clips and voice recordings, we show that we can convert the content of a given sequence into another one by such content swapping. For audio, this allows us to convert a male speaker into a female speaker and vice versa, while for video we can separately manipulate shapes and dynamics. Furthermore, we give empirical evidence for the hypothesis that stochastic RNNs as latent state models are more efficient at compressing and generating long sequences than deterministic ones, which may be relevant for applications in video compression.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “One can also exploit known relations between factors in different samples (Karaletsos et al., 2015; Goroshin et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton and Birodkar, 2017; Hsu et al., 2017; Yingzhen and Mandt, 2018b; Locatello et al., 2018).”
Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, background): “[67] introduce a stochastic latent variable model for sequential data which also yields disentangled representations and allows content swapping between generated sequences.”
On the Fairness of Disentangled Representations (2019, background): “ 46, 42, 43, 44, 32]. Disentangling pose from content and content from motion are also classical computer vision problems that have been tackled with various degrees of supervision and inductive bias [92, 93, 40, 25, 18, 31, 42]. In this paper, we intend disentanglement in the sense of [6, 87, 35, 6”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “35, 37, 55, 49]. [20, 10] assume some knowledge of the effect of the factors of variations even though they are not observed. One can also exploit known relations between factors in different samples [18,52,16,13,21,54]. This is not a limiting assumption especially in sequential data, i.e., for vide”
Recent Advances in Autoencoder-Based Representation Learning (2018, methodology): “ere is a considerable number of recent works leveraging (variational) autoencoders and the techniques similar to those outlined in Sections3–5to learn representations of sequences. Yingzhen and Mandt [68] partition the latent code of a VAE into subsets of time varying and time invariant variables (r”
Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning (2019, background): “…based on variational inference (Chung et al., 2015; Rezende et al., 2014; Kingma and Welling, 2014), including (Babaeizadeh et al., 2018; Denton and Fergus, 2018; Lee et al., 2018; Li and Mandt, 2018; He et al., 2018) and large-scale models such as (Castrejon et al., 2019; Villegas et al., 2019).”
Unsupervised Acoustic Unit Representation Learning for Voice Conversion using WaveNet Auto-encoders (2020, background): “In some recent voice conversion systems [3, 4, 5], autoencoder based models with disentanglement learning objective functions have achieved good performance.”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, unknown): “The Bund, Shanghai, with the ship moving on the river, oil painting.”
Disentangling Correlated Speaker and Noise for Speech Synthesis via Data Augmentation and Adversarial Factorization (2019, background): “Learning disentangled representations for deep generative models has gain much interest recently [9, 29].”
A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning (2020, methodology): “They have been used for disentangled representation learning in speech [13, 14, 15].”
GP-VAE: Deep Probabilistic Time Series Imputation (2019, methodology): “We performed experiments on the benchmark data set Healing MNIST [26], which combines the classical MNIST data set [29] with properties common to medical time series, the SPRITES data set [31], and on a real-world medical data set from the 2012 Physionet Challenge [43].”
Deep Probabilistic Video Compression (2018, methodology): “This model is refined to additionally include a global state f similar to Li & Mandt (2018), resulting in the following probabilistic deep generative model: pθ(x1:T , z1:T ,f) = pθ(f)pθ(z1:T ) T∏ t=1 pθ(xt | zt,f) (1) Above, θ is shorthand for the parameters.”
Time-series Generative Adversarial Networks (2019, background): “Other works have studied the utility of learning latent representations for purposes of pre-training [24], disentanglement [25], and interpretability [26].”
Dynamical Variational Autoencoders: A Comprehensive Review (2020, methodology): “n be computed, without sampling from any random variable. Once this is achieved, then samples from z t 1 as well as from z t are used to compute the t-th KL divergence term in (11.24). 2Note that in (Li and Mandt, 2018), Appendix A, the schematic representation of the inference model given in Fig. 9”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, background): “Several papers have investigated masked prediction [8, 86, 7, 103], while others have implemented autoencoder models with various constraints applied to the latent representations, such as discreteness [98, 25, 11] or disentanglement [50, 48, 65, 60].”
Diffusion Probabilistic Modeling for Video Generation (2022, background): “A large body of video generation research relies on variational deep sequential latent variable models [13,28,29,48–56], to name a few.”