Debiased Contrastive Learning

By Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, Stefanie Sabrina Jegelka. 2020

739 citations recorded by the source corpus.

A prominent technique for self-supervised representation learning has been to contrast semantically similar and dissimilar pairs of samples. Without access to labels, dissimilar (negative) points are typically taken to be randomly sampled datapoints, implicitly accepting that these points may, in reality, actually have the same label. Perhaps unsurprisingly, we observe that sampling negative examples from truly different labels improves performance, in a synthetic setting where labels are available. Motivated by this observation, we develop a debiased contrastive objective that corrects for the sampling of same-label datapoints, even without knowledge of the true labels. Empirically, the proposed objective consistently outperforms the state-of-the-art for representation learning in vision, language, and reinforcement learning benchmarks. Theoretically, we establish generalization bounds for the downstream classification task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Improving Text-to-Image Synthesis Using Contrastive Learning (2021, background): “In the last two years, various contrastive methods [2, 3, 5, 8, 9, 10, 15, 16, 21, 26, 28, 34, 35, 39] for visual representations have been proposed.”
Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE (2021, background): “These efforts range from theoretical investigations such as generalization error analyses [21] and asymptotic characterizations [22], to more practical aspects including hard-negative reinforcement [23, 24], and sampling bias adjustment [25].”
Self-Supervised Representation Learning: Introduction, advances, and challenges (2021, neutral): “When this assumption is violated, they suffer from false-positive pretext task labels [45].”
Hard Negative Mixing for Contrastive Learning (2020, background): “A few recent works discuss issues around the selection of negatives in contrastive self-supervised learning [4, 11, 23, 45, 47, 22].”
Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo (2022, background): “A key property of CL is that it involves negative samples, and a major line works [3,10,26,28,29,35,39,45,48,52,60] have shown that mining hard negative samples can be beneficial for performance.”
Contrastive Code Representation Learning (2020, background): “an image can generate diverse variants of a base image. However, text-based domains like code lack such simple transformations. The InfoNCE pre-training objective has been applied to natural language [54, 16], but is outperformed by baselines or requires auxiliary supervised models for data augmenta”
Contrastive Learning for Cold-Start Recommendation (2021, background): “The contrastive learning is widely used in the field of computer vision [4, 6, 17] and natural language processing [5, 26, 27].”
AAVAE: Augmentation-Augmented Variational Autoencoders (2021, positive): “Considering a ﬂurry of recent work reporting on the importance of mining better negative examples [55, 11, 52], our approach based on autoencoding greatly simpliﬁes self-supervised learning by entirely eliminating negative examples.”
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing (2023, background): “However, the success of VLP relies on paired samples sharing semantics, i.e., given an image and text pair, the text should describe the image with minimal extraneous detail [15,16,35].”
Self-Supervised Learning on Graphs: Contrastive, Generative, or Predictive (2021, background): “Themost common sampling strategy is uniform sampling, but this has been shown to be very informative [85], [86], [87].”
Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, background): “der performance by drawing negative samples that are too similar, including more false-negatives [17].”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, neutral): “Several studies have delved into this area [54], [71], [72], [73], [74], [75], [76], [77], [78], [79], [80], [81].”
Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding (2021, background): “However, this assumption can suffer from the problem of sampling bias, common in most contrastive learning approaches (Chuang et al., 2020; Saunshi et al., 2019).”
Text Transformations in Contrastive Self-Supervised Learning: A Review (2022, background): “This sampling bias would lead to a sub-optimal representation as it cannot capture the true semantic structure of the data [Li et al., 2020; Chuang et al., 2020].”
Contrastive Learning with Hard Negative Samples (2020, methodology): “Our proposed hard negative sampling method conditions on the event { h ( x ) (cid:54) = h ( x − ) } in order to avoid false negatives (termed “debiasing” (Chuang et al., 2020)).”