Whitening for Self-Supervised Representation Learning

By Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, Nicu Sebe. 2020

400 citations recorded by the source corpus.

Recent literature on self-supervised learning is based on the contrastive loss, where image instances which share the same semantic content ("positives") are contrasted with instances extracted from other images ("negatives"). However, in order for the learning to be effective, a lot of negatives should be compared with a positive pair. This is not only computationally demanding, but it also requires that the positive and the negative representations are kept consistent with each other over a long training period. In this paper we propose a different direction and a new loss function for self-supervised learning which is based on the whitening of the latent-space features. The whitening operation has a "scattering" effect on the batch samples, which compensates the lack of a large number of negatives, avoiding degenerate solutions where all the sample representations collapse to a single point. We empirically show that our loss accelerates self-supervised training and the learned representations are much more effective for downstream tasks than previously published work.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “Several other works echo this direction, showing that one can train features matching them to a uniform distribution on the `2 hypersphere [5] or by using whitening [19, 66].”
Barlow Twins: Self-Supervised Learning via Redundancy Reduction (2021, methodology): “Alternatively, the redundancy reduction term can be viewed as a soft-whitening constraint on the representations, connecting our method to a recently proposed method performing a hard-whitening operation on the representations (Ermolov et al., 2020), as discussed in Section 5.”
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (2021, methodology): “The expanders have 3 fully-connected layers of size 8192. alternative class of collapse prevention methods relies on maximizing the information content of the embedding Zbontar et al. (2021); Ermolov et al. (2021).”
Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo (2022, background): “However, they are often dependent on additional predictor [19] or stop gradient [8] or explicit de-centering and de-correlation [4, 15, 60, 62].”
Context Autoencoder for Self-supervised Representation Learning (2022, neutral): “Self-supervised representation learning has been widely studied in computer vision (Dosovitskiy et al., 2014; 2015; Doersch et al., 2015; Xie et al., 2016; Yang et al., 2016; van den Oord et al., 2018; Caron et al., 2018; Asano et al., 2019; Caron et al., 2019; Huang et al., 2019; Zhuang et al., 201”
Decoupled Contrastive Learning (2021, background): “Other recent works [37,11] aim to mitigate the issue of small batch size in InfoNCE loss.”
Efficient Training of Visual Transformers with Small Datasets (2021, background): “Most current self-supervised computer vision approaches can be categorised in contrastive learning [54, 25, 9, 23, 52, 56, 18], clustering methods [3, 67, 29, 5, 1, 20, 6, 7], asymmetric networks [22, 10] and feature-decorrelation methods [19, 63, 2, 27].”
Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning (2021, methodology): “…2021), MoCo V2+ (Chen et al., 2020b), NNCLR (Dwibedi et al., 2021), ReSSL (Zheng et al., 2021), SimCLR (Chen et al., 2020a), Supervised Contrastive Learning (Khosla et al., 2020), SimSiam (Chen and He, 2021), SwAV (Caron et al., 2020), VICReg (Bardes et al., 2021) and W-MSE (Ermolov et al., 2021).”
On Feature Decorrelation in Self-Supervised Learning (2021, methodology): “For medium-size datasets STL-10 and Tiny ImageNet, only the max-pooling layer is disabled following [14, 10].”
A Cookbook of Self-Supervised Learning (2023, methodology): “Mialon et al. [2022] argue that the projector enforces pairwise independence of the features in the representation and provide a demonstration for random projectors in the context of VICReg, BarlowTwins and W-MSE [Bardes et al., 2021, Zbontar et al., 2021, Ermolov et al., 2021].”
Joint Learning of Localized Representations from Medical Images and Reports (2021, background): “In recent years, contrastive approaches[85,59,32,29,53,43,7,28,27,8,89,3,20], have become the state-of-the-art discriminative approaches for self-supervised representation learning.”
Un-mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning (2020, methodology): “... codebase conﬁgurations [Ermolov et al. 2020b] with 500 epochs using the Adam optimizer without any data augmentation.”
From Canonical Correlation Analysis to Self-supervised Graph Neural Networks (2021, background): “W-MSE [8] performs a differentiable whitening operation on learned embeddings, which implicitly scatters data points in embedding space.”
On the Representation Collapse of Sparse Mixture of Experts (2022, background): “Besides, this phenomenon has also been observed in language model ﬁne-tuning (Aghajanyan et al., 2021), and visual representation learning (Chen and He, 2021; Ermolov et al., 2021; Jing et al., 2022).”
Covariance Structure and Coordinate Heterogeneity Govern Binary Quantization of Contrastive Embeddings (2026, unknown): “Isotropy-promoting methods (Ermolov et al., 2021; Bardes et al., 2022) regularize toward uniform variances but do not analyze what happens when residual heterogeneity remains.”