Hard Negative Mixing for Contrastive Learning

By Yannis Kalantidis, Mert Bulent Sariyildiz, No'e Pion, Philippe Weinzaepfel, Diane Larlus. 2020

775 citations recorded by the source corpus.

Contrastive learning has become a key component of self-supervised learning approaches for computer vision. By learning to embed two augmented versions of the same image close to each other and to push the embeddings of different images apart, one can train highly transferable visual representations. As revealed by recent studies, heavy data augmentation and large sets of negatives are both crucial in learning such representations. At the same time, data mixing strategies either at the image or the feature level improve both supervised and semi-supervised learning by synthesizing novel examples, forcing networks to learn more robust features. In this paper, we argue that an important aspect of contrastive learning, i.e., the effect of hard negatives, has so far been neglected. To get more meaningful negative samples, current top contrastive self-supervised learning approaches either substantially increase the batch sizes, or keep very large memory banks; increasing the memory size, however, leads to diminishing returns in terms of performance. We therefore start by delving deeper into a top-performing framework and show evidence that harder negatives are needed to facilitate better and faster learning. Based on these observations, and motivated by the success of data mixing, we propose hard negative mixing strategies at the feature level, that can be computed on-the-fly with a minimal computational overhead. We exhaustively ablate our approach on linear classification, object detection and instance segmentation and show that employing our hard negative mixing procedure improves the quality of visual representations learned by a state-of-the-art self-supervised learning method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Improving Text-to-Image Synthesis Using Contrastive Learning (2021, background): “In the last two years, various contrastive methods [28, 2, 9, 3, 16, 35, 26, 5, 8, 10, 15, 21, 34, 39] for visual representations have been proposed.”
Support-set bottlenecks for video-text representation learning (2020, background): “, 2020b) and feature mixing (Kalantidis et al., 2020) which have been applied in the image domain.”
MixCo: Mix-up Contrastive Learning for Visual Representation (2020, methodology): “Such representations are highly transferable to down-stream tasks [5, 12, 19], and even outperform the representations learned by the supervised manner in some tasks [7, 35].”
Un-mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning (2020, background): “…Hjelm et al. 2018, Gidaris, Singh, and Komodakis 2018, He et al. 2019, Chen et al. 2020a, Kim et al. 2020, Grill et al. 2020, Caron et al. 2020, Kalantidis et al. 2020] due to its enormous potential of being free from human-annotated supervision, i.e. , its extraordinary capability of…”
Training GANs with Stronger Augmentations via Contrastive Discriminator (2021, background): “…“fake” samples for contrastive learning, which is also an interesting direction along with other recent attempts to improve the efﬁciency of negative sampling in contrastive learning, e . g ., via hard negative mining (Kalantidis et al., 2020; Robinson et al., 2021). for t = 1 to N D do 3:”
Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE (2021, background): “These efforts range from theoretical investigations such as generalization error analyses [21] and asymptotic characterizations [22], to more practical aspects including hard-negative reinforcement [23, 24], and sampling bias adjustment [25].”
OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning (2020, background): “Among them, contrastive-based instance discrimination training [9, 18, 33, 40, 71] is the most prominent example.”
Decoupled Mixup for Data-efficient Learning (2022, neutral): “, 2021), and are utilized in contrastive learning to learn discriminative visual representation (Kalantidis et al., 2020; Lee et al., 2021; Shen et al., 2021; Li et al., 2021b).”
When and why vision-language models behave like bags-of-words, and what to do about it? (2022, background): “4 A SIMPLE FIX : C OMPOSITION - AWARE HARD NEGATIVES Our analysis of the retrieval objective and datasets leads to a natural solution: hard negatives for contrastive learning (Robinson et al., 2021; Kalantidis et al., 2020).”
Text Transformations in Contrastive Self-Supervised Learning: A Review (2022, background): “In the context of vision, Kalantidis [Kalantidis et al., 2020] also raised the same issue with the in-batch negative sampling as well as the time-consuming use of memory banks that needs to keep a large memory up-to-date.”
Neighborhood Contrastive Learning for Novel Class Discovery (2021, background): “Note that this is in contrast with the recent literature on hard negative mining [18, 25], which samples hard negatives, incurring in the problem of false-negatives.”
On Feature Decorrelation in Self-Supervised Learning (2021, neutral): “…decorrelation with representa-method batch size top-1 InstDisc [46] 256 58.5 LocalAgg [52] 128 58.8 MoCo [19] 256 60.6 SimCLR [8] 256 61.9 CPC v2 [35] 512 63.8 PCL v2 [33] 256 67.6 MoCo v2 [9] 256 67.5 MoCHi [29] 512 68.0 PIC [4] 512 67.6 AdCo [24] 256 68.6 Shuffled-DBN 512 65.18 tive methods.”
Towards Domain-Agnostic Contrastive Learning (2020, methodology): “as ours and (Shen et al., 2020; Kalantidis et al., 2020; Kim et al., 2020b) have advantage over recently proposed adversarial direction based contrastive learning method (Kim et al.”
MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation (2021, background): “Unlike Kalantidis et al. (2020), we do not determine hardness of negative pairs based on similarities of representations.”
SCEHR: Supervised Contrastive Learning for Clinical Risk Prediction using Electronic Health Records (2021, background): “The majority of contrastive learning literature adopted self-supervised techniques [22], [23], [31], [32] by building augmented data with pseudo-labels.”
Self-Supervised Learning for Recommender Systems: A Survey (2022, background): “Feature Mixing [80], [65] mixes the original user/item features with features from other users/items or previous versions to synthesize informative negative/positive examples [81].”
Decoupled Contrastive Learning (2021, background): “[18] and [25] learn better representation by sampling hard negatives, particularly for small batches.”
i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning (2021, background): “Kalantidis et al. (2020) mixed hard negative samples on the embedding space.”
A Survey on Contrastive Self-supervised Learning (2020, background): “[85] proposed a few hard negative mixing strategies to facilitate faster and better learning.”