Barlow Twins: Self-Supervised Learning via Redundancy Reduction

By Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, Stéphane Deny. 2021

3,100 citations recorded by the source corpus.

Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. A successful approach to SSL is to learn embeddings which are invariant to distortions of the input sample. However, a recurring issue with this approach is the existence of trivial constant solutions. Most current methods avoid such solutions by careful implementation details. We propose an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible. This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors. The method is called Barlow Twins, owing to neuroscientist H. Barlow's redundancy-reduction principle applied to a pair of identical networks. Barlow Twins does not require large batches nor asymmetry between the network twins such as a predictor network, gradient stopping, or a moving average on the weight updates. Intriguingly it benefits from very high-dimensional output vectors. Barlow Twins outperforms previous methods on ImageNet for semi-supervised classification in the low-data regime, and is on par with current state of the art for ImageNet classification with a linear classifier head, and for transfer tasks of classification and object detection.

Subfields: Artificial Intelligence, Biophysics, Media Technology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Media Technology

In Representation & Self-Supervised Learning

Citations in the corpus

Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “Several other works echo this direction, showing that one can train features matching them to a uniform distribution on the `2 hypersphere [5] or by using whitening [19, 66].”
High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, neutral): “The latest, most successful, modern family of SSL approaches for images (Misra & Maaten, 2020; Chen et al., 2020; Chen & He, 2020; He et al., 2020; Grill et al., 2020; Caron et al., 2020; 2021; Zbontar et al., 2021; Bardes et al., 2021), have two noteworthy characteristics that markedly distinguish ”
Towards Label-Efficient Incremental Learning: A Survey (2023, methodology): “Also, self-supervision for incremental learning solely relied on contrastive learning-based approaches to extract supervisory signals [Chen et al. , 2020; Zbontar et al. , 2021 ] .”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, neutral): “…[6] 79.75 ± 0.84* DyTox [14] 79.21 ± 0.10 L2P [66] 81.07 ± 0.13* EWC [32] 0/class 49.19 SimCLR [10]-CaSSLe [18] 0/class 44.2* BYOL [20]-CaSSLe [18] 49.7* Barlow Twins[72]-CaSSLe [18] 48.9* Supervised Contrastive [30]-CaSSLe [18] 50.9* L2P [66] 40 image/language prompting for S-iPrompts/S-liPrompts.”
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, positive): “As shown in Table 4, CoSCL can significantly improve the performance of UCL with two strong unsupervised learning strategies such as SimSiam [8] and BarlowTwins [51].”
Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation (2022, methodology): “The proposed Continual Barlow Twins (CBT) algorithm trains a feature extractor (ResNet50) with Barlow Twins (BT) [3], whose loss is integrated with a regularization term, borrowed from Elastic Weight Consolidation (EWC) [8], to avoid catastrophic forget-”
The Challenges of Continuous Self-Supervised Learning (2022, positive): “Prior work has developed improved image augmentations [ 12 , 46 ], backbone models [ 10 , 24 ], stable (slow-moving) learning targets [ 9 , 15 , 29 ], and transformation invariant loss functions [ 10 , 13 , 27 , 56 , 84 ].”
Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, neutral): “As outlined by the field of contrastive self-supervised learning [43], [44], the skillful use of data augmentation techniques can lead towards useful representations even when no label information is made available to the learner.”
Representational Continuity for Unsupervised Continual Learning (2021, methodology): “In this work, we focus on SimSiam (Chen & He, 2021) and BarlowTwins (Zbontar et al., 2021), which tackle these limitations and achieve state-of-the-art performance on standard representation learning benchmarks.”
How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, positive): “1 INTRODUCTION Recent advances in self-supervised learning (SSL) (He et al., 2020; Grill et al., 2020; Caron et al., 2020; Jure et al., 2021) demonstrate competitive or even better transfer learning performance on various downstream tasks, compared with supervised pre-training.”
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (2021, methodology): “…(2020) 66.7 64.9 SwAV Caron et al. (2020) 65.7 62.7 Barlow Twins Zbontar et al. (2021) 64.8 62.9 VICReg 64.5 62.8 VICReg borrows the decorrelation mechanism of Barlow Twins Zbontar et al. (2021) and we observe that it therefore has the same dependency on the dimensionality of the expander network.”
Masked Siamese Networks for Label-Efficient Learning (2022, background): “Our goal is to learn image representations by first pre-training on D and then adapting the representation to the supervised task using S. Siamese Networks The goal of siamese networks (Becker & Hinton, 1992; Bromley et al., 1993), as they are used in self-supervised learning, is to learn an encoder”
On the Difficulty of Defending Self-Supervised Learning against Model Extraction (2022, background): “Barlow Twins (Zbontar et al., 2021) aims to create similar representations of two augmentations of the same image while reducing redundancy between the output units of the representations.”
Graph Self-Supervised Learning: A Survey (2021, methodology): “Apart from BYOL, Barlow Twins [86] is another similar yet powerful method without using negative samples to prevent the model from collapsing.”
Rethinking the Representational Continuity: Towards Unsupervised Continual Learning (2021, methodology): “In this work, we focus on SimSiam (Chen & He, 2021) and BarlowTwins (Zbontar et al., 2021), which tackle these limitations and achieve state-of-the-art performance on standard representation learning benchmarks.”
A Cookbook of Self-Supervised Learning (2023, methodology): “This key element of data-augmentation had also been largely used in the following SSL works [Chen et al., 2020d, Bardes et al., 2021, Zbontar et al., 2021] without significant changes.”