Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction

By Richard Zhang, Phillip Isola, Alexei A. Efros. 2016

698 citations recorded by the source corpus.

We propose split-brain autoencoders, a straightforward modification of the traditional autoencoder architecture, for unsupervised representation learning. The method adds a split to the network, resulting in two disjoint sub-networks. Each sub-network is trained to perform a difficult task – predicting one subset of the data channels from another. Together, the sub-networks extract features from the entire input signal. By forcing the network to solve cross-channel prediction tasks, we induce a representation within the network which transfers well to other, unseen tasks. This method achieves state-of-the-art performance on several large-scale transfer learning benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neurology

In Representation & Self-Supervised Learning

Citations in the corpus

Momentum Contrast for Unsupervised Visual Representation Learning (2019, neutral): “, denoising auto-encoders [58], context autoencoders [48], or cross-channel auto-encoders (colorization) [64, 65].”
Learning Visual Representations with Caption Annotations (2020, background): “tasks consist in predicting missing pieces in manipulated images, for instance context prediction [15], colorization [13,36,66], inpainting of missing portions [46], prediction of image rotations [19], spotting artifacts [31], or cross-channel prediction [67].”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Pretext tasks are defined on unlabeled images in the hopes that solving them requires some nontrivial semantic understanding; examples including solving jigsaw puzzles [47], predicting rotation [19], colorization [17, 18], inpainting [16], context prediction [15], clustering [27], and generative mod”
Contrastive Multiview Coding (2019, background): “ different types of mutual information across views. 4.4. Predictive Learning vs. Contrastive Learning? While experiments in section 4.1 show that contrastive learning outperforms predictive learning [63] in the context of Lab color space, it’s unclear whether such an advantage is due to the natural”
Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, methodology): “We compare our method with a randomly initialized network (as a lower bound) and various unsupervised learning methods, including self-supervised learning [2, 47, 27, 48], adversarial learning [4], and Exemplar CNN [3].”
Unsupervised Representation Learning by Predicting Image Rotations (2018, background): “Speciﬁcally, we ﬁrst train a RotNet model on the training images of the ImageNet dataset and then we evaluate the performance of the self-supervised features on the image Context (Doersch et al., 2015) 19.7 26.7 31.9 32.7 30.9 Context Encoders (Pathak et al., 2016b) 18.2 23.2 23.4 21.9 18.4 Coloriza”
Learning Features by Watching Objects Move (2016, background): “Other pretext tasks include predicting color channels from luminance [25, 51] or vice versa [52], and predicting sounds from video frames [7,33].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “Prior work suggests that the hyperparameters used in transfer learning can play an important role in the evaluation of pre-trained representations [21, 35, 87].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, methodology): “1 LINEAR PROBES AND BASELINE ARCHITECTURE In order to quantify if a neural network has learned useful feature representations, we follow the standard approach of using linear probes (Zhang et al., 2017).”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, methodology): “Examples include context prediction [14], colorization of grayscale images [14], inpainting of missing portions of images [56], predicting rotations of images [20], spotting artifacts of images [36], and the SplitBrain method [76].”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, methodology): “The main remedy suggested in the literature is to introduce Method Accuracy Context [24] 0.317 BiGAN [34, 36] 0.310 Colorization [25] 0.326 RotNet [26] 0.387 DeepClustering [35] [15] counter discriminator forgetting by leveraging techniques from continual learning directly (Elastic Weight Sharing…”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, methodology): “Most existing self-supervised approaches [12, 75] use ImageNet-1k for pre-training and evaluate the representations on the same dataset.”
CrossTransformers: spatially-aware few-shot transfer (2020, background): “Our work on SimCLR episodes inherits from a line of self-supervised learning research, which typically deal with transfer from pretext tasks to semantic ones and must therefore represent more than their training data [3, 13, 15, 21, 24, 30, 36, 47, 60, 101, 102].”
Deep Clustering for Unsupervised Learning of Visual Features (2018, methodology): “[72], we train a linear classifier on top of different frozen convolutional layers.”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Many self-supervised methods manipulate the input data to extract a supervised signal in the form of a pretext task [1, 13, 26, 28, 30, 36, 38, 41, 42, 47, 48, 57].”
Representation Learning by Learning to Count (2017, methodology): “For example, some methods recover color from gray scale images and vice versa [43, 21, 44, 22], recover a whole patch from the surrounding pixels [33], or recover the relative location of patches [9, 29].”
What makes for good views for contrastive learning (2020, methodology): “We follow [53, 64] to split each color space into two views, such as {Y,DbDr} and {R,GB}.”