VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

By Adrien Bardes, Jean Ponce, Yann LeCun. 2021

1,331 citations recorded by the source corpus.

Recent self-supervised methods for image representation learning are based on maximizing the agreement between embedding vectors from different views of the same image. A trivial solution is obtained when the encoder outputs constant vectors. This collapse problem is often avoided through implicit biases in the learning architecture, that often lack a clear justification or interpretation. In this paper, we introduce VICReg (Variance-Invariance-Covariance Regularization), a method that explicitly avoids the collapse problem with a simple regularization term on the variance of the embeddings along each dimension individually. VICReg combines the variance term with a decorrelation mechanism based on redundancy reduction and covariance regularization, and achieves results on par with the state of the art on several downstream tasks. In addition, we show that incorporating our new variance term into other methods helps stabilize the training and leads to performance improvements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, neutral): “…(Misra & Maaten, 2020; Chen et al., 2020; Chen & He, 2020; He et al., 2020; Grill et al., 2020; Caron et al., 2020; 2021; Zbontar et al., 2021; Bardes et al., 2021), have two noteworthy characteristics that markedly distinguish them from traditional unsupervised-learning models such as…”
On Generalizing Beyond Domains in Cross-Domain Continual Learning (2022, methodology): “[1] apply covariances for correlation and decorrelation among samples to avoid collapse (i.”
Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning (2021, methodology): “For example, [3, 14, 35] rely on feature decorrelation to solve the mode collapse problem in self-supervised learning when negative pairs are missing, and [5,15,34] use decorrelation to improve neural networks’ features and hence boost generalization.”
Self-Supervised Models are Continual Learners (2021, methodology): “For VICReg [3] we only consider the invariance term.”
Self-Supervised Learning in Remote Sensing: A review (2022, methodology): “Since redundancy-reduction-based algorithms (Barlow Twins [87] and VICReg [88]) are relatively new, there are only works using these methods in the remote sensing community.”
Contrastive Distillation Is a Sample-Efficient Self-Supervised Loss Policy for Transfer Learning (2022, positive): “The current paradigm is arguably one of "unsupervised pre-training then online supervised transfer", where the pre-training step is training on massive datasets using unsupervised losses like InfoNCE, language modeling loss, or VICReg, among others (Oord et al., 2019; Bardes et al., 2022), and the o”
Lossy Compression for Lossless Prediction (2021, background): “(7) from the main text) can be seen as contrastive SSL [15, 37] with an information bottleneck, a version of [109, 110] with an information instead of a variance bottleneck, a SSL VIB [92], or an invariant VAE [14].”
Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning (2021, methodology): “For Barlow Twins, BYOL, MoCo V2+, NNCLR, SimCLR and VICReg, hyperparameters were heavily tuned, reaching higher performance than reported on original papers or third-party results.”
Behavior Transformers: Cloning k modes with one stone (2022, result): “This is in stark contrast to vision and language tasks, where pretrained models and data-driven priors are the norm [19, 11, 32, 6], which allows for efﬁcient downstream task solving.”
Understanding Dimensional Collapse in Contrastive Self-supervised Learning (2021, methodology): “, 2018; 2020), redundancy reduction methods (Zbontar et al., 2021; Bardes et al., 2021) and methods using special architecture design (Grill et al.”
Context Autoencoder for Self-supervised Representation Learning (2022, neutral): “80], instance discrimination [28,27], image discretization [34,35], masked image modeling [59,31,74], and information maximization [30,97,5].”
Efficient Training of Visual Transformers with Small Datasets (2021, background): “Most current self-supervised computer vision approaches can be categorised in contrastive learning [54, 25, 9, 23, 52, 56, 18], clustering methods [3, 67, 29, 5, 1, 20, 6, 7], asymmetric networks [22, 10] and feature-decorrelation methods [19, 63, 2, 27].”
Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss (2021, background): “…learning, which trains representations on unlabeled data with surrogate losses and self-deﬁned supervision signals (Bachman et al., 2019, Bardes et al., 2021, Caron et al., 2020, Chen and He, 2020, Henaff, 2020, Hjelm et al., 2018, Misra and Maaten, 2020, Oord et al., 2018, Tian et…”
Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo (2022, background): “Our work fills the gap by pointing out that the loss in non-CL frameworks [4, 8, 15, 19, 59] treats anchors equally, while vanilla contrastive InfoNCE in CL penalizes each anchor based on their hardness.”
Joint Learning of Localized Representations from Medical Images and Reports (2021, background): “However, they are not suited for generative tasks and many aspects, like the need for negative sampling, are not well-understood yet although being tackled by approaches like BYOL[27], SimSiam[8], BarlowTwins[89], and VICReg[3].”
A Cookbook of Self-Supervised Learning (2023, methodology): “From these origins, stems SSL methods such as VICReg [Bardes et al., 2021], Barlow Twins [Zbontar et al.”
Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (2023, background): “gated to prevent representation collapse, such as contrastive losses that explicitly push apart embeddings of negative examples [15,24,37], non-contrastive losses that minimize the informational redundancy across embeddings [10, 74], and clustering-based approaches that maximize the entropy of the a”