Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies

By Alessandro Achille, Tom Eccles, Loic Matthey, Christopher P. Burgess, Nicholas Watters, Alexander Lerchner, Irina Higgins. 2018

136 citations recorded by the source corpus.

Intelligent behaviour in the real-world requires the ability to acquire new knowledge from an ongoing sequence of experiences while preserving and reusing past knowledge. We propose a novel algorithm for unsupervised representation learning from piece-wise stationary visual data: Variational Autoencoder with Shared Embeddings (VASE). Based on the Minimum Description Length principle, VASE automatically detects shifts in the data distribution and allocates spare representational capacity to new knowledge, while simultaneously protecting previously learnt representations from catastrophic forgetting. Our approach encourages the learnt representations to be disentangled, which imparts a number of desirable properties: VASE can deal sensibly with ambiguous inputs, it can enhance its own representations through imagination-based exploration, and most importantly, it exhibits semantically meaningful sharing of latents between different datasets. Compared to baselines with entangled representations, our approach is able to reason beyond surface-level statistics and perform semantically meaningful cross-domain inference.

Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “In recent years, interest in continual learning has risen [1, 21, 26, 27, 29, 39, 40, 42, 51], especially from the deep learning research community, due to its potential to reduce training time and training set sizes (e.g., by continuously adapting from previous models), both of which are critical…”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “[5]) to overcome catastrophic forgetting.”
DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “One possible approach would be to use Continual SRL methods like S-TRIGGER [Caselles-Dupré et al., 2019] or VASE [Achille et al., 2018].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Disentangled representations have been found to be more sample-efficient [29], less sensitive to nuisance variables [55], and better in terms of (systematic) generalization [1, 16, 28, 35, 70].”
Towards a Definition of Disentangled Representations (2018, background): “If moreover ρ 1 is the trivial representation that sends everything to the identity, then every g ∈ G will be represented by the matrix A = [1] , and ρ 1 ⊗ ρ 2 will have the same matrix as ρ 2 .”
Closed-Loop GAN for continual Learning (2018, background): “Currently, these models usually make temporary copies of the entire network to distill knowledge [Shin et al., 2017, Achille et al., 2018], but this requires complete retraining of the network.”
A soft nearest-neighbor framework for continual semi-supervised learning (2022, neutral): “While the vast majority of these methods operate in a supervised setting, recent works addressed the problem of overcoming catastrophic forgetting in the challenging case of limited or no supervision [1, 8, 21, 28, 38, 46, 47, 53].”
The Challenges of Continuous Self-Supervised Learning (2022, methodology): “The training data stream is created by sampling images at random from the four datasets {Dp1 , Dp2 , Dp3 , Dp4 } where [ p1 , p2 , p3 , p4] is a permutation of the sequence [1 , 2 , 3 , 4] .”
Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, background): “Recently, some attention has been devoted to latent regularization in continual image classification [1, 2, 27].”
Continual Learning with Adaptive Weights (CLAW) (2019, background): “…in continual learning (McCloskey & Cohen, 1989; Ratcliff, 1990; Robins, 1993; 1995; French, 1999; Pape et al., 2011; Goodfellow et al., 2014a; Achille et al., 2018; Kemker et al., 2018; Kemker & Kanan, 2018; Diaz-Rodriguez et al., 2018; Zeno et al., 2018; Ahn et al., 2019; Parisi et al.,…”
Continual Unsupervised Representation Learning (2019, background): “In contrast, Achille et al. (2018) address the problem of unsupervised learning in a sequential setting by learning a disentangled latent space with task-speciﬁc attention masks, but the main focus is on learning across datasets, and the method relies on abrupt shifts in data distribution between…”
Lifelong Generative Modeling (2017, methodology): “Contrast to VASE The recent work of Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies (VASE) [46] extend upon our work [46, p.”
Generative Models from the perspective of Continual Learning (2018, background): “A second method experimented on VAEs is to use a studentteacher method where the student learns the current task while the teacher retains knowledge [22].”
Continual Learning for Robotics (2019, background): “…models (cf. section 4) as adversarial models (Wu et al., 2018a; Lesort et al., 2018b), variational auto-encoders (Nguyen et al., 2017; Ramapuram et al., 2017; Achille et al., 2018; Farquhar and Gal, 2018; Lesort et al., 2018b) and standard auto-encoders (Triki et al., 2017; Zhou et al., 2012).”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, methodology): “Both continual generative modeling, Shin et al. (2017); Achille et al. (2018); Farquhar and Gal (2018a); Nguyen et al. (2018); Wu et al. (2018); Zhai et al. (2019), as well as the bulk of class incremental continuous learning works Li and Hoiem (2016); Kirkpatrick et al. (2017); Rebuﬃ et al.…”
Task-Agnostic Continual Learning Using Online Variational Bayes With Fixed-Point Updates (2020, methodology): “Little previous work is with taskagnostic scenarios (when task boundaries are unknown or not defined); however, they all use the rehearsal approach (Rao et al., 2019; Aljundi, Lin, Goujaud, & Bengio, 2019; Achille et al., 2018).”