With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations

By Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, Andrew Zisserman. 2021

552 citations recorded by the source corpus.

Self-supervised learning algorithms based on instance discrimination train encoders to be invariant to pre-defined transformations of the same instance. While most methods treat different views of the same image as positives for a contrastive loss, we are interested in using positives from other instances in the dataset. Our method, Nearest-Neighbor Contrastive Learning of visual Representations (NNCLR), samples the nearest neighbors from the dataset in the latent space, and treats them as positives. This provides more semantic variations than pre-defined transformations.We find that using the nearest-neighbor as positive in contrastive losses improves performance significantly on ImageNet classification using ResNet-50 under the linear evaluation protocol, from 71.7% to 75.6%, outperforming previous state-of-the-art methods. On semi-supervised learning benchmarks we improve performance significantly when only 1% ImageNet labels are available, from 53.8% to 56.5%. On transfer learning benchmarks our method outperforms state-of-the-art methods (including supervised learning with ImageNet) on 8 out of 12 downstream datasets. Furthermore, we demonstrate empirically that our method is less reliant on complex data augmentations. We see a relative reduction of only 2.1% ImageNet Top-1 accuracy when we train using only random crops.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Self-Supervised Models are Continual Learners (2021, methodology): “SimCLR[13] − log MoCo [28] NNCLR [19] InfoNCE DCV2 [7] DINO [8] Cross-entropy [58] u (1 Cross-correlation through the representations of the first argumentation.”
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm (2021, background): “Nearest-neighbor supervision Recently, researchers have exploited nearest-neighbor supervision to learn visual features (Dwibedi et al., 2021; Van Gansbeke et al., 2021).”
Efficient Training of Visual Transformers with Small Datasets (2021, background): “Most current self-supervised computer vision approaches can be categorised in contrastive learning [54, 25, 9, 23, 52, 56, 18], clustering methods [3, 67, 29, 5, 1, 20, 6, 7], asymmetric networks [22, 10] and feature-decorrelation methods [19, 63, 2, 27].”
Seed the Views: Hierarchical Semantic Alignment for Contrastive Representation Learning (2022, background): “Concurrent works NNCLR [29], [30] expand positives by searching nearest neighbors from an online-updated queue.”
Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning (2021, background): “For Barlow Twins, BYOL, MoCo V2+, NNCLR, SimCLR and VICReg, hyperparameters were heavily tuned, reaching higher performance than reported on original papers or third-party results.”
Decoupled Contrastive Learning (2021, methodology): “This leads to a plug-and-play improvement to the widely adopted InfoNCE-based contrastive learning; 4) We show that DCL can be easily combined with the SOTA contrastive methods, e.g. NNCLR [10], to achieve further improvements.”
Understanding Dimensional Collapse in Contrastive Self-supervised Learning (2021, unknown): “…2020b; He et al., 2020; Chen et al., 2020a; Chen & He, 2020; Grill et al., 2020; Zbontar et al., 2021; Bardes et al., 2021; Chen et al., 2020b; Dwibedi et al., 2021; Li et al., 2021; Misra & Maaten, 2020a; HaoChen et al., 2021; Assran et al., 2021; Caron et al., 2021) show that…”
A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision (2021, background): “DeCLIP [1] is another concurrent work that uses multiple unimodal SSL losses in addition to removing the noise from negatives with the help of nearest neighbors [11, 22].”
Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation (2021, methodology): “Instead of storing all feature vectors in the memory bank, we follow the same memory bank setting as in [5] which is for nearest neighbor retrieval.”
A Cookbook of Self-Supervised Learning (2023, background): “While with NNCLR [Dwibedi et al., 2021] the matched positive crop is replaced by its nearestneighbour in latent space, in MSF [Koohpayegani et al.”