On Feature Decorrelation in Self-Supervised Learning

By Tianyu Hua, Zihui Xue, Yue Wang, Sucheng Ren, Hang Zhao. 2021

244 citations recorded by the source corpus.

In self-supervised representation learning, a common idea behind most of the state-of-the-art approaches is to enforce the robustness of the representations to predefined augmentations. A potential issue of this idea is the existence of completely collapsed solutions (i.e., constant features), which are typically avoided implicitly by carefully chosen implementation details. In this work, we study a relatively concise framework containing the most common components from recent approaches. We verify the existence of complete collapse and discover another reachable collapse pattern that is usually overlooked, namely dimensional collapse. We connect dimensional collapse with strong correlations between axes and consider such connection as a strong motivation for feature decorrelation (i.e., standardizing the covariance matrix). The gains from feature decorrelation are verified empirically to highlight the importance and the potential of this insight.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning (2021, methodology): “For example, [3, 14, 35] rely on feature decorrelation to solve the mode collapse problem in self-supervised learning when negative pairs are missing, and [5,15,34] use decorrelation to improve neural networks’ features and hence boost generalization.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “The advancement of SSL [3], [4], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24] has exhibitedrapidprogress, capturing signiﬁcant attention within the research community (Fig.”
Consistency is the key to further mitigating catastrophic forgetting in continual learning (2022, background): “Despite the technical variety, most if not all SSL methods are prone to trivial, collapsed solutions by design (Hua et al., 2021).”
From Canonical Correlation Analysis to Self-supervised Graph Neural Networks (2021, background): “We show that the objective 1) essentially pursuits discarding augmentation-variant information and preserving augmentation-invariant information, and 2) can prevent dimensional collapse [19] (i.”
Understanding Dimensional Collapse in Contrastive Self-supervised Learning (2021, background): “This is known as dimensional collapse (Hua et al., 2021), whereby the embedding vectors only span a lower-dimensional subspace.”
Efficient Training of Visual Transformers with Small Datasets (2021, background): “Most current self-supervised computer vision approaches can be categorised in contrastive learning [54, 25, 9, 23, 52, 56, 18], clustering methods [3, 67, 29, 5, 1, 20, 6, 7], asymmetric networks [22, 10] and feature-decorrelation methods [19, 63, 2, 27].”