Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
By Jean-Bastien Grill, Florian Strub, Florent Altch'e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, B. '. Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar and 4 more. 2020
8,797 citations recorded by the source corpus.
We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\%$ top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and $79.6\%$ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation and pretrained models are given on GitHub.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Sergey Levine
- Dacheng Tao
- Trevor Darrell
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Representation & Self-Supervised Learning
- Trevor Darrell
- Yoshua Bengio
- Ishan Misra
- Yann LeCun
- Dacheng Tao
- Andrea Vedaldi
- Andrew Zisserman
- Ruslan Salakhutdinov
Citations in the corpus
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “BYOL We use the recently released model weights of BYOL (Grill et al., 2020), specifically their 50x1 and 200x2 checkpoints.”
- Masked Autoencoders Are Scalable Vision Learners (2021, background): “Contrastive and related methods strongly depend on data augmentation [7, 20, 8].”
- Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “In efforts to make more efficient use of unlabeled image data, several recent studies have shown promising results from contrastive representation learning from natural images (Chen et al., 2020a; He et al., 2020; Grill et al., 2020).”
- BEiT: BERT Pre-Training of Image Transformers (2021, neutral): “BYOL [GSA+20] and SimSiam [CH20] further eliminate the requirement of negative samples, using various techniques to avoid representation collapse.”
- Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “We do not use a predictor [23, 14], resulting in the exact same architecture in both student and teacher networks.”
- An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “the big convolutional ResNets in prior art [10, 17].”
- Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “Recently, methods based on contrastive learning [21, 38] have shown that unsupervised pretraining produces features that surpass the supervised feature representations on many downstream tasks [7, 8, 20, 22, 37].”
- Grafit: Learning fine-grained image representations with coarse labels (2020, methodology): “To deal with finer semantic levels than those provided by the labels, we use an approach similar to BYOL [25].”
- Exploring Simple Siamese Representation Learning (2020, methodology): “We use batch normalization (BN) [22] synchronized across devices, following [8, 15, 7].”
- Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “In computer vision, in contrast, most recent unsupervised or self-supervised learning research focuses on applying different random augmentations to images, with the pretraining objective to distinguish image instances (Chen et al., 2020b; He et al., 2020; Chen et al., 2020d; Grill et al., 2020; Che”
- D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Contrastive self-supervised learning (SSL) methods can greatly reduce the need for labeled data in discriminative tasks by learning effective representations from unlabeled data [95, 37, 35], and have also been shown to improve few-shot learning [39].”
- CogView: Mastering Text-to-Image Generation via Transformers (2021, background): “(Paul Cézanne [17]) As contrastive self-supervised pretraining has revolutionized computer vision (CV) [24, 21, 8, 32], visual-language pretraining, which brings high-level semantics to images, is becoming the next frontier of visual understanding [38, 30, 39].”
- SLIP: Self-supervision meets Language-Image Pre-training (2021, methodology): “Recent methods for self-supervised learning also propose a variety of alternatives to the contrastive objective such as self-distillation [25, 6], or input reconstruction [1, 26].”
- High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, background): “The latest, most successful, modern family of SSL approaches for images (Misra & Maaten, 2020; Chen et al., 2020; Chen & He, 2020; He et al., 2020; Grill et al., 2020; Caron et al., 2020; 2021; Zbontar et al., 2021; Bardes et al., 2021), have two noteworthy characteristics that markedly distinguish…”
- CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, background): “The CLIP model is considered as an important foundation model (Bommasani et al., 2021), therefore a plethora of follow-up work has been published (see Appendix Section A.4).”
- ScaleNet: An Unsupervised Representation Learning Method for Limited Information (2023, methodology): “MoCo [20], SimCLR [6], BYOL [18], SIMSIAM [8] as contrastive learning methods, modify a two-branch network architectures to generate two different augmentations of an image and maximize the similarity outputs from two branches.”
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, background): “sentations are often learned through instance classification, in which each image and its augmentations are treated as a single output class to be pulled together [14], [15] or contrasted against other negative samples [22].”
- Scaling Vision Transformers (2021, methodology): “In this paper we concentrate on scaling laws for transfer performance of ViT models pre-trained on image classifica- ?equal contribution 1 105 25 Examples per class 60 65 70 75 80 85 Im ag eN et to p- 1 ac cu ra cy Our ViT-G ViT-H SimCLR v2 BYOL DINO Figure 1.”