Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation

By Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, Baining Guo. 2022

146 citations recorded by the source corpus.

Masked image modeling (MIM) learns representations with remarkably good fine-tuning performances, overshadowing previous prevalent pre-training approaches such as image classification, instance contrastive learning, and image-text alignment. In this paper, we show that the inferior fine-tuning performance of these pre-training approaches can be significantly improved by a simple post-processing in the form of feature distillation (FD). The feature distillation converts the old representations to new representations that have a few desirable properties just like those representations produced by MIM. These properties, which we aggregately refer to as optimization friendliness, are identified and analyzed by a set of attention- and optimization-related diagnosis tools. With these properties, the new representations show strong fine-tuning performance. Specifically, the contrastive self-supervised learning methods are made as competitive in fine-tuning as the state-of-the-art masked image modeling (MIM) algorithms. The CLIP models' fine-tuning performance is also significantly improved, with a CLIP ViT-L model reaching 89.0% top-1 accuracy on ImageNet-1K classification. On the 3-billion-parameter SwinV2-G model, the fine-tuning accuracy is improved by +1.5 mIoU / +1.1 mAP to 61.4 mIoU / 64.2 mAP on ADE20K semantic segmentation and COCO object detection, respectively, creating new records on both benchmarks. More importantly, our work provides a way for the future research to focus more effort on the generality and scalability of the learnt representations without being pre-occupied with optimization friendliness since it can be enhanced rather easily. The code will be available at https://github.com/SwinTransformer/Feature-Distillation.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment (2022, methodology): “Only pre-training the ViT-Huge on ImageNet-1K [8], our model can outperform other methods with larger models (e.g., BEIT-3 [27] and SwinV2-G [18, 29]) and more pre-training data.”
A Cookbook of Self-Supervised Learning (2023, methodology): “It has been shown that contrasting methods show inferior performance than masked image modeling with regards to fine-tuning, because they are less “optimization friendly” [Wei et al., 2022] - which explains the overall interest over MIM.”
ImageBind One Embedding Space to Bind Them All (2023, methodology): “Feature Alignment Pre-trained CLIP models have been utilized as teachers to supervise other models due to the strength of its visual representations [44, 58, 75].”
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale (2022, methodology): “(timestamp: Nov 10, 2022) methods / results reference. a: BEiT-3 [104], b: iBOT [130], c: Open CLIP-H [48], d: Text4Vis [112], e: MaskFeat [106], f: Group DETRv2 [19], g: FocalNet [119], h: FD-SwinV2-G [110], i: Mask DINO [57], j: LVIS 2021 competition 1 st [36], k: ViT-Adapter [23]. pure MIM pre-tr”
EVA-02: A Visual Representation for Neon Genesis (2023, unknown): “9tta -tta FD-SwinV2-G [127] ∼3000M -tta -tta 64.”
Focal Modulation Networks (2022, methodology): “It also outperforms the SwinV2-G tuned with the new feature distillation technique [87], and establishes a new SoTA on the COCO leaderboard.”
Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning (2022, background): “Work on this problem has led to significant improvements along several threads including image level predictions like image classification [13, 54, 58], object level predictions like object detection [33, 51, 53,59–61], as well as pixel level predictions like semantic segmentation [10,29,51,53].”