Recent Advances in Autoencoder-Based Representation Learning

By Michael Tschannen, Olivier Bachem, Mario Lucic. 2018

511 citations recorded by the source corpus.

Learning useful representations with little or no supervision is a key challenge in artificial intelligence. We provide an in-depth review of recent advances in representation learning with a focus on autoencoder-based models. To organize these results we make use of meta-priors believed useful for downstream tasks, such as disentanglement and hierarchical organization of features. In particular, we uncover three main mechanisms to enforce such properties, namely (i) regularizing the (approximate or aggregate) posterior distribution, (ii) factorizing the encoding and decoding distribution, or (iii) introducing a structured prior distribution. While there are some promising results, implicit or explicit supervision remains a key enabler and all current methods use strong inductive biases and modeling assumptions. Finally, we provide an analysis of autoencoder-based representation learning through the lens of rate-distortion theory and identify a clear tradeoff between the amount of prior knowledge available about the downstream tasks, and how useful the representation is for this task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Sometimes generative models are also evaluated using linear classiﬁcation (Radford et al., 2016; Donahue et al., 2017; Dumoulin et al., 2017), or semi-supervised learning (Kingma et al., 2014; Narayanaswamy et al., 2017; Tschannen et al., 2018).”
A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “A recent line of work has argued that representations that are disentangled are an important step towards a better representation learning (Bengio et al., 2013; Peters et al., 2017; LeCun et al., 2015; Bengio et al., 2007; Schmidhuber, 1992; Lake et al., 2017; Tschannen et al., 2018).”
On Mutual Information Maximization for Representation Learning (2019, background): “…et al. (2018) similarly observed that models achieving the same evidence lower bound value can lead to vastly different representations depending on the employed encoder architecture, and do not necessarily capture useful information about the data (Tschannen et al., 2018; Blau and Michaeli, 2019).”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “The key underlying assumption is that high-dimensional observations x (such as images or videos) are in fact a manifestation of a low-dimensional set of independent ground-truth factors of variation z (Locatello et al., 2019b; Bengio et al., 2013; Tschannen et al., 2018).”
Learning Representations by Maximizing Mutual Information in Variational Autoencoders (2019, background): “VAEs have also found use as a representation learning model for inferring latent variables [9].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, methodology): “Learning good representations of high-dimensional sensory data is of fundamental importance to Artiﬁcial Intelligence [4, 3, 6, 48, 7, 67, 65, 49, 57, 71].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “A recent line of work has argued that representations that are disentangled are an important step towards a better representation learning (Bengio et al., 2013; Peters et al., 2017; LeCun et al., 2015; Bengio et al., 2007; Schmidhuber, 1992; Lake et al., 2017; Tschannen et al., 2018).”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, background): “…of disentangle-ment learning: i) the lack of a formal notion of disentangled representations to support the design of proper objective functions [42, 26], and ii) the lack of robust evaluation metrics to enable a fair comparison between models, regardless of their architectures or design…”
Self-Supervised Learning: Generative or Contrastive (2020, background): “On the other hand, achieving the same lower bound value can lead to vastly different representations and performance on downstream tasks, which indicates that it does not necessarily capture useful information of data [3] [128] [13].”
Disentangling Factors of Variation Using Few Labels (2019, methodology): “We refer to Appendix B of [42] and Section 3 of [62] for a more detailed description of these regularizers.”
A Unifying Review of Deep and Shallow Anomaly Detection (2020, background): “The manifold assumption is also widespread in related unsupervised learning tasks, such as manifold learning itself [353], [354], dimensionality reduction [3], [355]–[357], disentanglement [210], [358], and representation learning, in general [80], [359].”
Deep learning in medical image registration: a review (2019, background): “Examples of regularized autoencoders include sparse autoencoder, denoising autoencoder and contractive autoencoder [151].”
Face identity disentanglement via latent space mapping (2020, background): “A properly disentangled representation can benefit semantic data mixing [Johnson et al. 2016; Xiao et al. 2019], transfer learning for downstream tasks [Bengio et al. 2013; Tschannen et al. 2018], or even interpretability [Mathieu G Mapping 𝐸 𝐸 (cid:3033)̅ 𝑤 ∈ 𝒲 𝑧 Fig.”
In the Wild: From ML Models to Pragmatic ML Systems (2020, methodology): “Unsupervised learning methods like autoencoders [43, 33] and more recent self-supervision methods [21] like Momentum Contrast (MoCo) [18] and SimCLR [4] have begun to produce representations that rival that of supervised learning and achieve similar accuracies on various tasks.”
Disentangling Influence: Using Disentangled Representations to Audit Model Predictions (2019, background): “This approach has been very successful in generating representations in deep learning that can be manipulated while creating realistic inputs [3, 4, 8, 16, 25].”
A Survey on Deep Learning for Multimodal Data Fusion (2020, background): “More autoencoder variants can be found in Michael et al. (2018).”
ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection (2018, methodology): “To prevent the net from discarding precious information during training, we rely on autoencoder-based representation learning [48] by which the latent space is constrained to preserve all the data necessary to reconstruct the image in compact form.”
The Visual Task Adaptation Benchmark (2019, background): “Sometimes generative models are also evaluated using linear classification (Radford et al., 2016; Donahue et al., 2017; Dumoulin et al., 2017), or semi-supervised learning (Kingma et al., 2014; Narayanaswamy et al., 2017; Tschannen et al., 2018).”