BYOL works even without batch statistics

By Pierre H. Richemond, Jean-Bastien Grill, Florent Altch'e, Corentin Tallec, Florian Strub, Andrew Brock, Samuel L. Smith, Soham De, Razvan Pascanu, Bilal Piot and 1 more. 2020

131 citations recorded by the source corpus.

Bootstrap Your Own Latent (BYOL) is a self-supervised learning approach for image representation. From an augmented view of an image, BYOL trains an online network to predict a target network representation of a different augmented view of the same image. Unlike contrastive methods, BYOL does not explicitly use a repulsion term built from negative pairs in its training objective. Yet, it avoids collapse to a trivial, constant representation. Thus, it has recently been hypothesized that batch normalization (BN) is critical to prevent collapse in BYOL. Indeed, BN flows gradients across batch elements, and could leak information about negative views in the batch, which could act as an implicit negative (contrastive) term. However, we experimentally show that replacing BN with a batch-independent normalization scheme (namely, a combination of group normalization and weight standardization) achieves performance comparable to vanilla BYOL ($73.9\%$ vs. $74.3\%$ top-1 accuracy under the linear evaluation protocol on ImageNet with ResNet-$50$). Our finding disproves the hypothesis that the use of batch statistics is a crucial ingredient for BYOL to learn useful representations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “Of particular importance, our framework is flexible and works on both convnets and ViTs without the need to modify the architecture, nor adapt internal normalizations [47].”
Barlow Twins: Self-Supervised Learning via Redundancy Reduction (2021, background): “, 2020) or alternatively group normalization (Richemond et al., 2020) could play an important role in avoiding collapse.”
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (2021, methodology): “…batch-wise or feature-wise normalization, a "momentum encoder" in which the parameter vector of one branch is a low-pass-ﬁltered version of the parameter vector of the other branch Grill et al. (2020); Richemond et al. (2020), or a stop-gradient operation in one of the branches Chen & He (2020).”
Seed the Views: Hierarchical Semantic Alignment for Contrastive Representation Learning (2022, methodology): “[25] demonstrates that negative samples are not necessary for contrastive learning with the asymmetry network architecture, it is also classiﬁed as a feature reconstruction method in [26].”
On Feature Decorrelation in Self-Supervised Learning (2021, background): “Tian et al . provide an analysis of how various factors involved in BYOL and SimSiam work together to prevent collapse [42].”
Unlocking High-Accuracy Differentially Private Image Classification through Scale (2022, background): “…can be an eﬀective replacement for batch normalization in non-private training, especially when training with large batch size (Kolesnikov et al., 2020; Qiao et al., 2019; Richemond et al., 2020), while Brock et al. (2021a) found Weight Standardization improves the performance of NF-ResNets.”
AAVAE: Augmentation-Augmented Variational Autoencoders (2021, background): “BYOL [29, 51, 56], and its simpler variant called SimSIAM [9], rely on asymmetry in the network architecture between the teacher and student to avoid such degeneracy.”
Whitening for Self-Supervised Representation Learning (2020, background): “The reason why this does not happen is still an open problem and this has opened a debate in the community (Fetterman & Albrecht, 2020; Tian et al., 2020b; Chen & He, 2020; Richemond et al., 2020).”
Bootstrapped Representation Learning on Graphs (2021, methodology): “For this task, we found the use of layer normalization (Ba et al., 2016) and weight standardization (Qiao et al., 2019) to be more effective than batch normalization at stabilizing training, following an alternative stabilizing technique used in Richemond et al. (2020).”