Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo

By Chaoning Zhang, Kang Zhang, Trung X. Pham, Axi Niu, Zhinan Qiao, Chang-Dong Yoo, In-So Kweon. 2022

74 citations recorded by the source corpus.

Contrastive learning (CL) is widely known to require many negative samples, 65536 in MoCo for instance, for which the performance of a dictionary-free framework is often inferior because the negative sample size (NSS) is limited by its mini-batch size (MBS). To decouple the NSS from the MBS, a dynamic dictionary has been adopted in a large volume of CL frameworks, among which arguably the most popular one is MoCo family. In essence, MoCo adopts a momentum-based queue dictionary, for which we perform a fine-grained analysis of its size and consistency. We point out that InfoNCE loss used in MoCo implicitly attract anchors to their corresponding positive sample with various strength of penalties and identify such inter-anchor hardness-awareness property as a major reason for the necessity of a large dictionary. Our findings motivate us to simplify MoCo v2 via the removal of its dictionary as well as momentum. Based on an InfoNCE with the proposed dual temperature, our simplified frameworks, Sim-MoCo and SimCo, outperform MoCo v2 by a visible margin. Moreover, our work bridges the gap between CL and non-CL frameworks, contributing to a more unified under-standing of these two mainstream frameworks in SSL. Code is available at: https://bit.ly/3LkQbaT.

Subfields: Artificial Intelligence, Civil and Structural Engineering, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. In-So Kweon
2. Tom Goldstein
3. Andrew Gordon Wilson
4. Yuandong Tian
5. Yann LeCun
6. Micah Goldblum
7. Hamed Pirsiavash
8. Ruslan Salakhutdinov

In Civil and Structural Engineering

1. Micah Goldblum
2. Andrew Gordon Wilson
3. Yann LeCun
4. Tom Goldstein
5. Hamed Pirsiavash
6. In-So Kweon
7. Ari S. Morcos
8. Jonas Geiping

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Jitendra Malik
4. Xiaogang Wang
5. Trevor Darrell
6. Ming-Hsuan Yang
7. Kristen Grauman
8. Andrew Zisserman

In Representation & Self-Supervised Learning

1. In-So Kweon
2. Hamed Pirsiavash
3. Tom Goldstein
4. Adrien Bardes
5. Yuandong Tian
6. Grégoire Mialon
7. Yann LeCun
8. Florian Bordes

Citations in the corpus

• CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, neutral): “Yeh et al. (2021) and Zhang et al. (2022) reported similar observations of overfitting.”
• A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, background): “Later, the trend of discriminative visual SSL shifts from such geometry-based prediction to joint-embedding emthods [19], [20], [21].”
• A Cookbook of Self-Supervised Learning (2023, background): “Recent works have modified BYOL [Grill et al., 2020] or Barlow Twins [Zbontar et al., 2021] by feeding images of consecutive timesteps’ observations to the two branches of the siamese network: SGI [Schwarzer et al., 2021b] and Barlow Balance [Zhang et al., 2022c] did this for offline pretraining, wh”