Understanding Dimensional Collapse in Contrastive Self-supervised Learning
By Li Jing, Pascal Vincent, Yann LeCun, Yuandong Tian. 2021
496 citations recorded by the source corpus.
Self-supervised visual representation learning aims to learn useful representations without relying on human annotations. Joint embedding approach bases on maximizing the agreement between embedding vectors from different views of the same image. Various methods have been proposed to solve the collapsing problem where all embedding vectors collapse to a trivial constant solution. Among these methods, contrastive learning prevents collapse via negative sample pairs. It has been shown that non-contrastive methods suffer from a lesser collapse problem of a different nature: dimensional collapse, whereby the embedding vectors end up spanning a lower-dimensional subspace instead of the entire available embedding space. Here, we show that dimensional collapse also happens in contrastive learning. In this paper, we shed light on the dynamics at play in contrastive learning that leads to dimensional collapse. Inspired by our theory, we propose a novel contrastive learning method, called DirectCLR, which directly optimizes the representation space without relying on an explicit trainable projector. Experiments show that DirectCLR outperforms SimCLR with a trainable linear projector on ImageNet.
Subfields: Artificial Intelligence, Media Technology, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Furu Wei
- Dacheng Tao
- Li Dong
- Jianfeng Gao
- Mohit Bansal
- Luke S. Zettlemoyer
- Graham Neubig
- Michael Auli
In Media Technology
- Dacheng Tao
- Hamed Pirsiavash
- Adrien Bardes
- Andrew Gordon Wilson
- Jing Zhang
- Yann LeCun
- Tom Goldstein
- Randall Balestriero
In Representation & Self-Supervised Learning
- Alexei Baevski
- Michael Auli
- Wei-Ning Hsu
- Dacheng Tao
- Qiantong Xu
- Furu Wei
- In-So Kweon
- Hamed Pirsiavash
Citations in the corpus
- data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “This occurs when the model produces very similar representations for all masked segments which results in a trivial task (Jing et al., 2021).”
- A Cookbook of Self-Supervised Learning (2023, background): “, 2022a], the classical rank estimator [Jing et al., 2022], fitting a power law to the singular value distribution [Ghosh et al.”
- On the Representation Collapse of Sparse Mixture of Experts (2022, methodology): “, 2021), and visual representation learning (Chen and He, 2021; Ermolov et al., 2021; Jing et al., 2022).”
- A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “Numerous CL-based methods are available in the literature [88], [89], [90], [91], [92], [93], [94], [95], [96].”
- A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, background): “Later, the trend of discriminative visual SSL shifts from such geometry-based prediction to joint-embedding emthods [19], [20], [21].”