Dense Contrastive Learning for Self-Supervised Visual Pre-Training

By Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, Lei Li. Computer Vision and Pattern Recognition, 2020

842 citations recorded by the source corpus.

To date, most existing self-supervised learning methods are designed and optimized for image classification. These pre-trained models can be sub-optimal for dense prediction tasks due to the discrepancy between image-level prediction and pixel-level prediction. To fill this gap, we aim to design an effective, dense self-supervised learning method that directly works at the level of pixels (or local features) by taking into account the correspondence between local features. We present dense contrastive learning (DenseCL), which implements self-supervised learning by optimizing a pairwise contrastive (dis)similarity loss at the pixel level between two views of input images.Compared to the baseline method MoCo-v2, our method introduces negligible computation overhead (only <1% slower), but demonstrates consistently superior performance when transferring to downstream dense prediction tasks including object detection, semantic segmentation and instance segmentation; and outperforms the state-of-the-art methods by a large margin. Specifically, over the strong MoCo-v2 baseline, our method achieves significant improvements of 2.0% AP on PASCAL VOC object detection, 1.1% AP on COCO object detection, 0.9% AP on COCO instance segmentation, 3.0% mIoU on PASCAL VOC semantic segmentation and 1.8% mIoU on Cityscapes semantic segmentation.Code and models are available at: https://git.io/DenseCL

View Paper: https://www.semanticscholar.org/paper/6f92dcefc5f6b4346f619ae7546a8bd2d6decade

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations (2022, methodology): “In MC-16, the DenseCL [72] pre-trained backbone improves its performance by 11.4 with our synthetic dataset.”
EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (2023, background): “One line of work is contrastive learning methods[9, 11, 57, 62], which learn augmentation in-variance by imposing high similarity be-tween different augmented views of a given image.”
Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo (2022, background): “Especially, contrastive learning (CL) frameworks [2,6,22,23,25,36,44,47,50,53,65] have attracted significant attention due to its intuitive motivation.”
Exploring Cross-Image Pixel Contrast for Semantic Segmentation (2021, background): “Though a few concurrent works also address contrastive learning in dense image prediction [75, 7, 69], the ideas are significantly different.”
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting (2021, methodology): “We consider supervised pre-training on ImageNet1K [11] and ImageNet21K [11, 36], self-supervised pre-training via MoCoV2 [16] and DenseCL [43], and the vision-language pre-training.”
Context Autoencoder for Self-supervised Representation Learning (2022, unknown): “Some dense variants (Wang et al., 2021; Xie et al., 2021a) eliminate the tendency in a limited degree by considering an extra contrastive loss with dense patches.”
Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective (2022, methodology): “, visual categorization [15, 17, 33], object detection [14, 31, 40], and semantic segmentation [25, 46].”
Bootstrapping Semantic Segmentation with Regional Contrast (2021, methodology): “Contrastive pre-training based on dense representations has recently been explored, and shows better performance in dense prediction tasks, such as object detection and keypoint detection (Wang et al., 2021b; O. Pinheiro et al., 2020).”
SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation (2022, methodology): “Recently, multi-view based methods such as MoCo [23], SimCLR [8], SimSiam [9] and BYOL [22] present better or comparable performances than supervised pretraining in transfer learning tasks including image classification, object detection [43] and semantic segmentation [41].”
iBOT: Image BERT Pre-Training with Online Tokenizer (2021, background): “Similar to Wang et al. (2021b), we match for each patch token fi the most similar patch token from another view f ′j , where j = argmaxj sim(fi, f ′ j). sim(u, v) is cosine distance.”
A Cookbook of Self-Supervised Learning (2023, methodology): “It is worth pointing out that some works have only noticed minor increases in performance [Wang et al., 2021a] where it only lead to a 0.3 point performance increase.”
Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (2021, methodology): “[58] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li.”
Joint Learning of Localized Representations from Medical Images and Reports (2021, background): “Therefore, there is a number of recent approaches that use region-level contrast[87,86,83,6,61,52], i.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “Various methods have been proposed to address this challenge, including the pixel-to-propagation consistency method [65] and dense contrastive learning [269].”
Efficient Self-supervised Vision Transformers for Representation Learning (2021, background): “Though the ideas of leveraging local-level pre-training tasks have been explored for ConvNets in the context of contrastive learning in a bid to improve dense visual prediction tasks [65, 60], our LR is different in that no negative samples/queue is needed, we find that LR is particularly effective ”
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation (2022, background): “…learning approaches such as: gray-scale image colorization [53], jigsaw puzzle solving [32], split-brain auto-encoding [54], rotation prediction [13], learning to cluster [2], or predicting values of some channels with one or two other channels as input [53, 54], and pixel-level contrast [44, 38].”
SODA10M: Towards Large-Scale Object Detection Benchmark for Autonomous Driving (2021, methodology): “For different downstream detection tasks listed in this table, the MoCo methods (MoCo-v1 [21], MoCo-v2 [5]) and dense contrastive methods (DenseCL [56], DetCo [58]) can achieve better results, while the other methods perform even worse than ImageNet fully supervised pre-train.”
DetCo: Unsupervised Contrastive Learning for Object Detection (2021, methodology): “InsLoc[41], DenseCL[39] and PatchReID[8] are recent works designed for object detection.”
TS2Vec: Towards Universal Representation of Time Series (2021, methodology): “Unsupervised Representation of Time Series Unsuper-vised representation learning has achieved good performances in computer vision (Chen et al. 2020; Wang et al. 2021; Xu et al. 2020; O. Pinheiro et al. 2020), natural language processing (Gao, Yao, and Chen 2021; Logeswaran and Lee 2018) and speech…”