Decoupled Contrastive Learning
By Chun-Hsiao Yeh, Cheng-Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, Yann LeCun. 2021
259 citations recorded by the source corpus.
Contrastive learning (CL) is one of the most successful paradigms for self-supervised learning (SSL). In a principled way, it considers two augmented"views"of the same image as positive to be pulled closer, and all other images as negative to be pushed further apart. However, behind the impressive success of CL-based techniques, their formulation often relies on heavy-computation settings, including large sample batches, extensive training epochs, etc. We are thus motivated to tackle these issues and establish a simple, efficient, yet competitive baseline of contrastive learning. Specifically, we identify, from theoretical and empirical studies, a noticeable negative-positive-coupling (NPC) effect in the widely used InfoNCE loss, leading to unsuitable learning efficiency concerning the batch size. By removing the NPC effect, we propose decoupled contrastive learning (DCL) loss, which removes the positive term from the denominator and significantly improves the learning efficiency. DCL achieves competitive performance with less sensitivity to sub-optimal hyperparameters, requiring neither large batches in SimCLR, momentum encoding in MoCo, or large epochs. We demonstrate with various benchmarks while manifesting robustness as much less sensitive to suboptimal hyperparameters. Notably, SimCLR with DCL achieves 68.2% ImageNet-1K top-1 accuracy using batch size 256 within 200 epochs pre-training, outperforming its SimCLR baseline by 6.4%. Further, DCL can be combined with the SOTA contrastive learning method, NNCLR, to achieve 72.3% ImageNet-1K top-1 accuracy with 512 batch size in 400 epochs, which represents a new SOTA in contrastive learning. We believe DCL provides a valuable baseline for future contrastive SSL studies.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Ocean Engineering, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Tom Goldstein
- In-So Kweon
- Andrew Gordon Wilson
- Yuandong Tian
- Quoc V. Le
- Sergey Levine
- Yann LeCun
- Yoshua Bengio
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Jitendra Malik
- Trevor Darrell
- Ming-Hsuan Yang
- Bernt Schiele
- Andrea Vedaldi
In Ocean Engineering
In Representation & Self-Supervised Learning
- In-So Kweon
- Hamed Pirsiavash
- Tom Goldstein
- Adrien Bardes
- Yuandong Tian
- Grégoire Mialon
- Sepp Hochreiter
- Yann LeCun
Citations in the corpus
- CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, background): “Yeh et al. (2021) and Zhang et al. (2022) reported similar observations of overfitting.”
- Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo (2022, background): “This rationale also somehow aligns with an observation in [56] that removing the positive pair from the denominator in the InfoNCE loss for decoupling the influence of positive sample and negative ones on each other.”
- Decoupled Mixup for Data-efficient Learning (2022, unknown): “A similar phenomenon has been recently described in contrastive learning [51] where the optimization of positive pairs reduces the training efficiency, which is termed as negative-positive-coupling.”
- A Cookbook of Self-Supervised Learning (2023, methodology): “Furthermore, some works such as DCL [Yeh et al., 2021] show that you can reach top performance with a batch size of 256 or more for SimCLR, and a queue size of only 256 or more for MoCo, by simply removing the positive pair from the denominator of the softmax and with more careful hyperparameter tun”