Contrastive Learning with Hard Negative Samples

By Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, Stefanie Sabrina Jegelka. 2020

1,052 citations recorded by the source corpus.

How can you sample good negative examples for contrastive learning? We argue that, as with metric learning, contrastive learning of representations benefits from hard negative samples (i.e., points that are difficult to distinguish from an anchor point). The key challenge toward using hard negatives is that contrastive methods must remain unsupervised, making it infeasible to adopt existing negative sampling strategies that use true similarity information. In response, we develop a new family of unsupervised sampling methods for selecting hard negative samples where the user can control the hardness. A limiting case of this sampling results in a representation that tightly clusters each class, and pushes different classes as far apart as possible. The proposed method improves downstream performance across multiple modalities, requires only few additional lines of code to implement, and introduces no computational overhead.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Improving Text-to-Image Synthesis Using Contrastive Learning (2021, background): “In the last two years, various contrastive methods [28, 2, 9, 3, 16, 35, 26, 5, 8, 10, 15, 21, 34, 39] for visual representations have been proposed.”
Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, background): “Furthermore, it is observed that negative examples, especially hard negative examples, benefit the learning of representation [37].”
Contrastive Learning Inverts the Data Generating Process (2021, background): “However, recent work has suggested some empirical evidence for said theoretical prediction, namely, issues with the commonly used sampling strategy for negative samples, and have proposed ways to mitigate said issues as well (Robinson et al., 2020; Chuang et al., 2020).”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “For instance, a hard negative sampling strategy [141] mines the negative pairs that are similar to the samples but likely belong to different classes.”
SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation (2022, background): “Besides, well aligned with previous works [15, 33] that claim "hard" positive pairs and negative pairs can boost the performance of contrastive learning, we can observe that larger magnitude (within an appropriate range) of the perturbation can bring consistent improvement of the performance.”
When and why vision-language models behave like bags-of-words, and what to do about it? (2022, background): “4 A SIMPLE FIX : C OMPOSITION - AWARE HARD NEGATIVES Our analysis of the retrieval objective and datasets leads to a natural solution: hard negatives for contrastive learning (Robinson et al., 2021; Kalantidis et al., 2020).”
Self-Supervised Change Detection in Multiview Remote Sensing Images (2021, background): “[42] proposed an effective hard negative sampling strategy to avoid the ”sampling bias” due to false-negative samples with same context information as the anchor.”
Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo (2022, background): “A key property of CL is that it involves negative samples, and a major line works [3,10,26,28,29,36,40,46,49,53,61] have shown that mining hard negative samples can be beneficial for performance.”
On Feature Decorrelation in Self-Supervised Learning (2021, background): “pairs [49, 2, 44] and calibrate the contribution of negative samples [44, 38, 27]; Another line of work uses clusteringbased mechanisms to characterize the relation of crosssample views [5, 6, 1, 7].”
Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approach (2022, background): “However, as with metric learning, contrastive learning beneﬁts from hard negatives (Cai et al., 2021; Robinson et al., 2020).”
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2022, background): “Based on that, ITM with hard negatives (ITMHard) is introduced to increase the difficulty of the task, which is shown to be very effective (Kalantidis et al., 2020; Robinson et al., 2021; Li et al., 2021b).”
One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2022, background): “However, negative examples have been shown to play an important role in contrastive learning (Robinson et al., 2021).”
A Cookbook of Self-Supervised Learning (2023, background): “Recently, some works have focused on putting more weight on hard samples [Robinson et al., 2020].”
Training GANs with Stronger Augmentations via Contrastive Discriminator (2021, background): “…“fake” samples for contrastive learning, which is also an interesting direction along with other recent attempts to improve the efﬁciency of negative sampling in contrastive learning, e . g ., via hard negative mining (Kalantidis et al., 2020; Robinson et al., 2021). for t = 1 to N D do 3:”
Neighborhood Contrastive Learning for Novel Class Discovery (2021, background): “Note that this is in contrast with the recent literature on hard negative mining [18, 25], which samples hard negatives, incurring in the problem of false-negatives.”
Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation (2024, background): “…objec-tive, which aims to instruct the model in prioritizing the generation of higher-quality translations and rejecting lesser ones, in a style of contrastive learning with hard negative examples (Oord et al., 2018; Chen et al., 2020; He et al., 2020; Robinson et al., 2021; Tan et al., 2023).”
Self-Supervised Learning on Graphs: Contrastive, Generative, or Predictive (2021, background): “Themost common sampling strategy is uniform sampling, but this has been shown to be very informative [85], [86], [87].”
Exploring Cross-Image Pixel Contrast for Semantic Segmentation (2021, background): “Negative pairs are usually randomly sampled, but some hard negative example mining strategies [41, 57, 38] were recently proposed.”
Decoupled Contrastive Learning (2021, background): “[18] and [25] learn better representation by sampling hard negatives, particularly for small batches.”