Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning

By Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, Han Hu. Computer Vision and Pattern Recognition, 2020

477 citations recorded by the source corpus.

Contrastive learning methods for unsupervised visual representation learning have reached remarkable levels of transfer performance. We argue that the power of contrastive learning has yet to be fully unleashed, as current methods are trained only on instance-level pretext tasks, leading to representations that may be sub-optimal for downstream tasks requiring dense pixel predictions. In this paper, we introduce pixel-level pretext tasks for learning dense feature representations. The first task directly applies contrastive learning at the pixel level. We additionally propose a pixel-to-propagation consistency task that produces better results, even surpassing the state-of-the-art approaches by a large margin. Specifically, it achieves 60.2 AP, 41.4 / 40.5 mAP and 77.2 mIoU when transferred to Pascal VOC object detection (C4), COCO object detection (FPN / C4) and Cityscapes semantic segmentation using a ResNet-50 backbone network, which are 2.6 AP, 0.8 / 1.0 mAP and 1.0 mIoU better than the previous best methods built on instance-level contrastive learning. Moreover, the pixel-level pretext tasks are found to be effective for pre-training not only regular backbone networks but also head networks used for dense downstream tasks, and are complementary to instance-level contrastive methods. These results demonstrate the strong potential of defining pretext tasks at the pixel level, and suggest a new path forward in unsupervised visual representation learning. Code is available at https://github.com/zdaxie/PixPro.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/74978168a087df56b67966aa57ead81fd79afe91

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (2023, background): “One line of work is contrastive learning methods[9, 11, 57, 62], which learn augmentation in-variance by imposing high similarity be-tween different augmented views of a given image.”
Seed the Views: Hierarchical Semantic Alignment for Contrastive Representation Learning (2022, methodology): “As shown in Table 11, CSML can also get comparable results with Pix-Pro [66] using the same conﬁguration.”
iBOT: Image BERT Pre-Training with Online Tokenizer (2021, methodology): “To compare, we include the results of supervised Swin-T (Liu et al., 2021b) which shares approximate parameter numbers with ViT-S/16 and its self-supervised counterpart MoBY (Xie et al., 2021a) in Tab.”
Self-Supervised Learning with Swin Transformers (2021, methodology): “In training, like most Transformer-based methods, we also adopt the AdamW [13, 15] optimizer, in contrast to previous self-supervised learning approaches built on ResNet backbone where usually SGD [9, 2] or LARS [4, 8, 19] is used.”
SpectralGPT: Spectral Remote Sensing Foundation Model (2023, background): “For example, some variants of SimCLR have incorporated new types of data augmentations or improved the training objective [16], [17], [18], while some variants of MoCo have explored different momentum update strategies or used additional losses to improve contrastive learning [19], [20], [21].”
Exploring Cross-Image Pixel Contrast for Semantic Segmentation (2021, background): “Though a few concurrent works also address contrastive learning in dense image prediction [75, 7, 69], the ideas are significantly different.”
Context Autoencoder for Self-supervised Representation Learning (2022, unknown): “Some dense variants [82,90] eliminate the tendency in a limited degree by considering an extra contrastive loss with dense patches.”
SimMIM: a Simple Framework for Masked Image Modeling (2021, background): “Another large portion of works lie in the contrastive learning approaches [3, 8, 16, 18, 20, 48, 50], which are the previous mainstream.”
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting (2021, methodology): “To alleviate the heavy annotation cost in previous supervised pre-training settings, a number of self-supervised pre-training approaches have been proposed for dense prediction [3, 43, 46, 47].”
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation (2022, background): “…learning approaches such as: gray-scale image colorization [53], jigsaw puzzle solving [32], split-brain auto-encoding [54], rotation prediction [13], learning to cluster [2], or predicting values of some channels with one or two other channels as input [53, 54], and pixel-level contrast [44, 38].”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “Various methods have been proposed to address this challenge, including the pixel-to-propagation consistency method [65] and dense contrastive learning [269].”
Joint Learning of Localized Representations from Medical Images and Reports (2021, methodology): “Therefore, there is a number of recent approaches that use region-level contrast[87,86,83,6,61,52], i.”