Understanding self-supervised Learning Dynamics without Contrastive Pairs

By Yuandong Tian, Xinlei Chen, Surya Ganguli. 2021

351 citations recorded by the source corpus.

While contrastive approaches of self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point (positive pairs) and maximizing views from different data points (negative pairs), recent \emph{non-contrastive} SSL (e.g., BYOL and SimSiam) show remarkable performance {\it without} negative pairs, with an extra learnable predictor and a stop-gradient operation. A fundamental question arises: why do these methods not collapse into trivial representations? We answer this question via a simple theoretical study and propose a novel approach, DirectPred, that \emph{directly} sets the linear predictor based on the statistics of its inputs, without gradient training. On ImageNet, it performs comparably with more complex two-layer non-linear predictors that employ BatchNorm and outperforms a linear predictor by $2.5\%$ in 300-epoch training (and $5\%$ in 60-epoch). DirectPred is motivated by our theoretical study of the nonlinear learning dynamics of non-contrastive SSL in simple linear networks. Our study yields conceptual insights into how non-contrastive SSL methods learn, how they avoid representational collapse, and how multiple factors, like predictor networks, stop-gradients, exponential moving averages, and weight decay all come into play. Our simple theory recapitulates the results of real-world ablation studies in both STL-10 and ImageNet. Code is released https://github.com/facebookresearch/luckmatters/tree/master/ssl.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Barlow Twins: Self-Supervised Learning via Redundancy Reduction (2021, methodology): “In contrast, our method avoids trivial solutions by construction, making our method conceptually simpler and more principled than these alternatives (until their principle is discovered, see (Tian et al., 2021) for an early attempt).”
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (2021, background): “The dynamics of learning in these methods, and how they avoid collapse, is not fully understood, although theoretical and empirical studies point to the crucial importance of batch-wise or feature-wise normalization Richemond et al. (2020); Tian et al. (2021).”
Self-Supervised Learning in Remote Sensing: A review (2022, background): “There are already some works, like SimSiam [84] and [249], that try to investigate the underlying theory, yet they are not enough.”
Continually Learning Self-Supervised Representations with Projected Functional Regularization (2021, background): “In the branch without predictor the gradient is not backpropagated during training, which was found to be crucial in preventing collapse to trivial solutions [55].”
Revisiting Feature Prediction for Learning Visual Representations from Video (2024, methodology): “· The use of an exponential-moving average feature extractor along with a stop-gradient and a predictor has been used as a collapse prevention strategy for image pre-training (Grill et al., 2020), and studied empirically (Xie et al., 2021) and theoretically (Tian et al., 2021).”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, neutral): “Tian et al. [254] studied the nonlinear learning dynamics of uncollated SSL in a simple linear network where SSL with only positive pairs avoids expression decay.”
On Feature Decorrelation in Self-Supervised Learning (2021, background): “provide an analysis of how various factors involved in BYOL and SimSiam work together to prevent collapse [42].”
Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style (2021, background): “[116] Yuandong Tian, Xinlei Chen, and Surya Ganguli.”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “negative-sample-free simple siamese (SimSiam [135]), Bootstrap (BYOL [136]), DirectPred [137]”
A Cookbook of Self-Supervised Learning (2023, background): “The predictor acts as a whitening operator preventing collapse [Tian et al., 2021], and momentum network can be applied only to the projector [Pham et al.”
From Canonical Correlation Analysis to Self-supervised Graph Neural Networks (2021, methodology): “We next provide a variance-covariance perspective to the new objective, following similar lines of reasoning in [41, 42].”
Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works (2022, background): “collapse resulting from the absence of negative samples [162].”
Pre-training Molecular Graph Representation with 3D Geometry (2021, background): “Stop-gradient For the optimization on variational representation reconstruction, related work have found that adding the stop-gradient operator (SG) as a regularizer can make the training more stable without collapse both empirically [11, 28] and theoretically [73].”
Large-Scale Representation Learning on Graphs via Bootstrapping (2021, neutral): “This non-collapsing behavior even without relying on negatives has been studied further (Tian et al., 2021).”