What Should Not Be Contrastive in Contrastive Learning

By Tete Xiao, Xiaolong Wang, Alexei A. Efros, Trevor Darrell. International Conference on Learning Representations, 2020

342 citations recorded by the source corpus.

Recent self-supervised contrastive methods have been able to produce impressive transferable visual representations by learning to be invariant to different data augmentations. However, these methods implicitly assume a particular set of representational invariances (e.g., invariance to color), and can perform poorly when a downstream task violates this assumption (e.g., distinguishing red vs. yellow cars). We introduce a contrastive learning framework which does not require prior knowledge of specific, task-dependent invariances. Our model learns to capture varying and invariant factors for visual representations by constructing separate embedding spaces, each of which is invariant to all but one augmentation. We use a multi-head network with a shared backbone which captures information across each augmentation and alone outperforms all baselines on downstream tasks. We further find that the concatenation of the invariant and varying spaces performs best across all tasks we investigate, including coarse-grained, fine-grained, and few-shot downstream classification tasks, and various data corruptions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c3a0059e69a10c8fe70efab423761e378fdab74b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Real-World Robot Learning with Masked Visual Pre-training (2022, methodology): “Simple and free from dataset or task-specific augmentations [43], MAE is the state95 of-the-art self-supervised framework in computer vision [44, 45, 46, 47], and has been demonstrated 96 to work well for motor control tasks in simulation [9].”
Graph Contrastive Learning with Adaptive Augmentation (2020, background): “[48], the data augmentation strategies should be adaptive to the input graph to reflect its intrinsic patterns.”
CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances (2020, background): “, which transformation should be used for positive or negative [66, 71].”
Contrastive Representation Learning: A Framework and Review (2020, background): “LooC [111] is an example where contrastive learning is used to concurrently learn multiple embedding sub-spaces, each of which is invariant to all but one transformation as specified by the distribution of positive pairs.”
Self-Supervised Pretraining Improves Self-Supervised Pretraining (2021, background): “In addition, instance contrastive learning is sensitive to the data augmentation policies and many trials are needed to determine the right settings [48, 60].”
Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, background): “A set of other work [2, 90, 110, 99, 91, 75, 95] focuses on providing empirical and theoretical understanding of contrastive learning.”
A Survey on Contrastive Self-supervised Learning (2020, background): “dependent aerial image recognition but might signiﬁcantly downgrade the performance while trying to solve downstream tasks such as detecting which way is up in a photograph for a display application. [25]. Similarly, colorization-based pretext tasks might not work out in a ﬁne-grain classiﬁcation re”
Self-Supervised Learning for Recommender Systems: A Survey (2022, neutral): “Althoughtherehavebeensomethe-oriesthattrytodemystifythevisualviewchoicesincontrastive learning[57],[137],theprincipleforaugmentationselectionin recommendationisseldomlystudied.”
Masked Feature Prediction for Self-Supervised Visual Pre-Training (2021, background): “, [16, 41]), which could lead to conflicting clues [71] and over-fitting to a specific combination of augmentations [94].”
Masked Visual Pre-training for Motor Control (2022, background): “Xiao et al. (2021b) have shown that the augmentations introduce inductive bias and may harm downstream transfer.”
A Cookbook of Self-Supervised Learning (2023, background): “Another line of work tries to retain style information about the augmentations to improve downstream performance on tasks requiring style information such as color by predicting style information [Xiao et al., 2020, Dangovski et al., 2021, Gidaris et al., 2018, Scherr et al., 2022].”
Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data (2021, methodology): “We get inspiration from [47] to develop a multi-augmentation contrastive learning method.”
Eigenfunction Extraction for Ordered Representation Learning (2025, unknown): “…and Isola, 2020; Wang et al., 2023; Huang et al., 2025; Wen and Li, 2021) and equivariance/invariance (Gupta et al., 2024; Garrido et al., 2023c; Xiao et al., 2021) focus on how representations evolve during training and how structural constraints, such as symmetry or transformation…”