Conditional Negative Sampling for Contrastive Learning of Visual Representations

By Mike Wu, Milan Mossé, Chengxu Zhuang, Daniel L. K. Yamins, Noah D. Goodman. 2020

91 citations recorded by the source corpus.

Recent methods for learning unsupervised visual representations, dubbed contrastive learning, optimize the noise-contrastive estimation (NCE) bound on mutual information between two views of an image. NCE uses randomly sampled negative examples to normalize the objective. In this paper, we show that choosing difficult negatives, or those more similar to the current instance, can yield stronger representations. To do this, we introduce a family of mutual information estimators that sample negatives conditionally -- in a "ring" around each positive. We prove that these estimators lower-bound mutual information, with higher bias but lower variance than NCE. Experimentally, we find our approach, applied on top of existing models (IR, CMC, and MoCo) improves accuracy by 2-5% points in each case, measured by linear evaluation on four standard image datasets. Moreover, we find continued benefits when transferring features to a variety of new image distributions from the Meta-Dataset collection and to a variety of downstream tasks such as object detection, instance segmentation, and keypoint detection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yifan Zhang
2. Jiashi Feng
3. Jian Liang
4. Dapeng Hu
5. Trung X. Pham
6. Chang-Dong Yoo
7. Zhinan Qiao
8. Bryan Hooi

In Computer Vision and Pattern Recognition

1. Jiashi Feng
2. Yifan Zhang
3. Jian Liang
4. Dapeng Hu
5. In-So Kweon
6. Chang-Dong Yoo
7. Bryan Hooi
8. Chaoning Zhang

In Representation & Self-Supervised Learning

1. Yifan Zhang
2. Jian Liang
3. Trung X. Pham
4. Chang-Dong Yoo
5. Zhinan Qiao
6. Bryan Hooi
7. Axi Niu
8. Kang Zhang

Citations in the corpus

• Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (2021, background): “Note that semi-hard negatives may even yield better performance in metric learning [61].”
• Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo (2022, background): “A key property of CL is that it involves negative samples, and a major line works [3,10,26,28,29,36,40,46,49,53,61] have shown that mining hard negative samples can be beneficial for performance.”