AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations Rather Than Data

By Liheng Zhang, Guo-Jun Qi, Liqiang Wang, Jiebo Luo. 2019

219 citations recorded by the source corpus.

The success of deep neural networks often relies on a large amount of labeled examples, which can be difficult to obtain in many real scenarios. To address this challenge, unsupervised methods are strongly preferred for training neural networks without using any labeled data. In this paper, we present a novel paradigm of unsupervised representation learning by Auto-Encoding Transformation (AET) in contrast to the conventional Auto-Encoding Data (AED) approach. Given a randomly sampled transformation, AET seeks to predict it merely from the encoded features as accurately as possible at the output end. The idea is the following: as long as the unsupervised features successfully encode the essential information about the visual structures of original and transformed images, the transformation can be well predicted. We will show that this AET paradigm allows us to instantiate a large variety of transformations, from parameterized, to non-parameterized and GAN-induced ones. Our experiments show that AET greatly improves over existing unsupervised approaches, setting new state-of-the-art performances being greatly closer to the upper bounds by their fully supervised counterparts on CIFAR-10, ImageNet and Places datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Contrastive Multiview Coding (2019, methodology): “A powerful family of models for unsupervised representations are collected under the umbrella of “self-supervised” learning (Sa, 2004; Zhang et al., 2017; 2016; Isola et al., 2015; Wang & Gupta, 2015; Pathak et al., 2016; Zhang et al., 2019).”
High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, methodology): “Pretext tasks for the image domain include solving Jigsaw-puzzles (Noroozi & Favaro, 2016), predicting rotations or afﬁne transformations (Gidaris et al., 2018; Zhang et al., 2019b) or discriminating instances (Wu et al., 2018; van den Oord et al., 2018).”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “The same is true for a range of other pretext tasks that have recently been studied [10, 20, 51, 85].”
What makes for good views for contrastive learning (2020, background): “Models based on contrastive losses have significantly outperformed other approaches [64, 30, 43, 53, 16, 41, 15, 19, 62].”
Prototypical Contrastive Learning of Unsupervised Representations (2020, background): “Examples include image inpainting [21], colorization [22, 23], prediction of patch orderings [24, 25] and image transformations [26, 27, 28, 29].”
Self-labelling via simultaneous clustering and representation learning (2019, background): “, 2016), predicting transformations (Gidaris et al., 2018; Zhang et al., 2019) and counting (Noroozi et al.”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, background): “xt tasks’ to learn visual representations. Pretext tasks are generally not useful by themselves and are used as a proxy to learn semantic representations. They can use the spatial structure in images [15,21,53,86], color [13,40,41,87], temporal information in videos [17,19,25,31,44,50,54,60,82] amon”
SEED: Self-supervised Distillation For Visual Representation (2021, background): “Major ramifications of visual SSL include pretext tasks (Noroozi & Favaro, 2016; Zhang et al., 2016; Gidaris et al., 2018; Zhang et al., 2019; Feng et al., 2019), contrastive representation learning (Wu et al.”
Unsupervised Pre-Training of Image Features on Non-Curated Data (2019, unknown): “This limits the use of a large number of cluster or a self-supervised task with a large output space [54].”
Rethinking Data Augmentation: Self-Supervision and Self-Distillation (2019, background): “The evidence that learning invariance is not always helpful could be found in many recent works on self-supervised learning (Gidaris et al., 2018; Zhang et al., 2019; Doersch et al., 2015; Noroozi & Favaro, 2016), where the model is trained with artiﬁcial labels constructed by input…”
Leveraging Large-Scale Uncurated Data for Unsupervised Pre-training of Visual Features (2019, background): “This limits the use of a large number of cluster or a self-supervised task with a large output space [54].”
Adversarial Fisher Vectors for Unsupervised Representation Learning (2019, background): “Compared with self-supervised learning approaches, such as [29, 30, 31, 32], our approach is density estimation based, and do not need any domain specific priors to create self supervision signal.”
Contrastive Learning With Stronger Augmentations (2021, methodology): “[20], [35] used the transformations between two images as supervised signals to guide the representation learning.”
An Incremental Self-Labeling Strategy for Semi-Supervised Deep Learning Based on Generative Adversarial Networks (2020, methodology): “Therefore, research on techniques such as efficient feature extraction [5], [6],”
WCP: Worst-Case Perturbations for Semi-Supervised Deep Learning (2020, background): “[34] combined multiple transformations to explore the self-supervised training of semi-supervised classifiers in an ensemble in an auto-encoding transformation fashion [35, 22, 15, 32] .”
Delving Deep Into Label Smoothing (2020, unknown): “Following AET [29], [30] and AVT [31], Wang et al. [32] proposed an innovative framework, EnAET [32], that combined semi-supervised and self-supervised training.”
Learning To Classify Images Without Labels (2020, background): “ature representations. The literature oers several pretext tasks which can be used to learn such an embedding function (e.g. rotation prediction [16], ane or perspective transformation prediction [57], colorization [29], in-painting [39], instance discrimination [51,18,7,32], etc.). In practice, ho”