Understanding the Behaviour of Contrastive Loss

By Feng Wang, Huaping Liu. 2020

894 citations recorded by the source corpus.

Unsupervised contrastive learning has achieved out-standing success, while the mechanism of contrastive loss has been less studied. In this paper, we concentrate on the understanding of the behaviours of unsupervised contrastive loss. We will show that the contrastive loss is a hardness-aware loss function, and the temperature τ controls the strength of penalties on hard negative samples. The previous study has shown that uniformity is a key property of contrastive learning. We build relations between the uniformity and the temperature τ. We will show that uniformity helps the contrastive learning to learn separable features, however excessive pursuit to the uniformity makes the contrastive loss not tolerant to semantically similar samples, which may break the underlying semantic structure and be harmful to the formation of features useful for downstream tasks. This is caused by the inherent defect of the instance discrimination objective. Specifically, instance discrimination objective tries to push all different instances apart, ignoring the underlying relations between samples. Pushing semantically consistent samples apart has no positive effect for acquiring a prior informative to general downstream tasks. A well-designed contrastive loss should have some extents of tolerance to the closeness of semantically similar samples. Therefore, we find that the contrastive loss meets a uniformity-tolerance dilemma, and a good choice of temperature can compromise these two properties properly to both learn separable features and tolerant to semantically similar samples, improving the feature qualities and the downstream performances.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “7Following past work, we optimize σ and clip it at 100, which enables the model to ‘warm-up’ its emphasis placed on hard negatives [91, 113].”
GLAZE: Protecting Artists from Style Mimicry by Text-to-Image Models (2023, methodology): “We test Glaze’s robustness to a stateof-the-art outlier detection method that leverages contrastive training [91].”
Knowledge Tracing: A Survey (2022, background): “, contrastive loss [115]) in a process called pre-training or pretext task [74] to automatically generate labels.”
Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive (2024, methodology): “Contrastive learning is widely used [Wang and Liu, 2021, Wang and Isola, 2020, Saunshi et al., 2019, Oord et al., 2018, Chen et al., 2020, He et al., 2020], often for embedding learning applications.”
Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (2022, background): “[46] show that low temperatures increase the model’s penalty on hard negative examples, and thus increase uniformity and decrease tolerance (the closeness of semantically similar samples).”
Self-Supervised Learning for Recommender Systems: A Survey (2022, background): “In addition, Wang and Liu [101] show that the InfoNCE loss is hardness-aware, and the temperature τ controls the strength of penalties on hard negative samples.”
Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo (2022, methodology): “Moreover, [47] has identified that InfoNCE has a hardness-aware property which is critical for competitive performance.”
Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (2021, methodology): “The potential reason is that a small temperature parameter implicitly helps the method to learn hard positive/negative pairs [55], which are more informative and beneficial to contrastive learning.”
Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings (2025, unknown): “Specifically, in contrastive learning, the temperature parameter is known to influence isotropy, with lower temperatures leading to more isotropic embeddings and higher temperatures resulting in less isotropic ones (Wang and Liu, 2021).”