Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding

By David A. Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, Dylan M. Paiton. 2020

160 citations recorded by the source corpus.

We construct an unsupervised learning model that achieves nonlinear disentanglement of underlying factors of variation in naturalistic videos. Previous work suggests that representations can be disentangled if all but a few factors in the environment stay constant at any point in time. As a result, algorithms proposed for this problem have only been tested on carefully constructed datasets with this exact property, leaving it unclear whether they will transfer to natural scenes. Here we provide evidence that objects in segmented natural movies undergo transitions that are typically small in magnitude with occasional large jumps, which is characteristic of a temporally sparse distribution. We leverage this finding and present SlowVAE, a model for unsupervised representation learning that uses a sparse prior on temporally adjacent observations to disentangle generative factors without any assumptions on the number of changing factors. We provide a proof of identifiability and show that the model reliably learns disentangled representations on several established benchmark datasets, often surpassing the current state-of-the-art. We additionally demonstrate transferability towards video datasets with natural dynamics, Natural Sprites and KITTI Masks, which we contribute as benchmarks for guiding disentanglement research towards more natural data domains.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, methodology): “Two of the baselines considered had hyperparameters to tune, SlowVAE (Klindt et al., 2021) and TCVAE (Chen et al., 2018).”
On the Transfer of Disentangled Representations in Realistic Settings (2020, methodology): “Recent work combines non-linear ICA with disentanglement (Khemakhem et al., 2020; Sorrenson et al., 2020; Klindt et al., 2020).”
Learnable latent embeddings for joint behavioural and neural analysis (2022, methodology): “We performed a sweep on perplexity in the range of [5, 50] and early_exaggeration in the range [12, 32] following the parameter guide, while fixing n_components as 2 and used a cosine metric, to fairly compare to UMAP and CEBRA.”
Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style (2021, background): “This development has sparked a renewed interest in identifiability in the context of deep representation learning [63, 64, 69, 83, 102, 108, 109, 129].”
Variable-rate discrete representation learning (2021, methodology): “More recently, Razavi et al. (2019a) and Klindt et al. (2021) used a probabilistic version (Turner & Sahani, 2007) to limit bottleneck capacity in variational autoencoders.”
Contrastive Learning Inverts the Data Generating Process (2021, methodology): “Recent work has generalized this contribution to VAEs (Khemakhem et al., 2020a; Locatello et al., 2020; Klindt et al., 2021), as well as (invertible-by-construction) energy-based models (Khemakhem et al.”