Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning

By Yifan Zhang, Bryan Hooi, Dapeng Hu, Jian Liang, Jiashi Feng. 2021

81 citations recorded by the source corpus.

Contrastive self-supervised learning (CSL) has attracted increasing attention for model pre-training via unlabeled data. The resulted CSL models provide instance-discriminative visual features that are uniformly scattered in the feature space. During deployment, the common practice is to directly fine-tune CSL models with cross-entropy, which however may not be the best strategy in practice. Although cross-entropy tends to separate inter-class features, the resulting models still have limited capability for reducing intra-class feature scattering that exists in CSL models. In this paper, we investigate whether applying contrastive learning to fine-tuning would bring further benefits, and analytically find that optimizing the contrastive loss benefits both discriminative representation learning and model optimization during fine-tuning. Inspired by these findings, we propose Contrast-regularized tuning (Core-tuning), a new approach for fine-tuning CSL models. Instead of simply adding the contrastive loss to the objective of fine-tuning, Core-tuning further applies a novel hard pair mining strategy for more effective contrastive fine-tuning, as well as smoothing the decision boundary to better exploit the learned discriminative feature space. Extensive experiments on image classification and semantic segmentation verify the effectiveness of Core-tuning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mingkui Tan
2. Yifan Zhang
3. Shuaicheng Niu
4. Chuang Gan
5. Jiashi Feng
6. Lanqing Hong
7. Zhenguo Li
8. Hongbin Lin

In Computer Vision and Pattern Recognition

1. Mingkui Tan
2. Jiashi Feng
3. Chuang Gan
4. Yifan Zhang
5. Zhenguo Li
6. Shuaicheng Niu
7. Lanqing Hong
8. Luc Van Gool

Citations in the corpus

• Prototype-Guided Continual Adaptation for Class-Incremental Unsupervised Domain Adaptation (2022, methodology): “…first attain source feature center f ks by: Then for any image prototype p i as an anchor, we conduct prototype alignment via the contrastive loss [57,19]: where v i = G ( p i ) denotes the extracted feature of the image prototype p i with ¯ y i as its pseudo label and τ denotes a temperature…”
• How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, neutral): “To achieve better performance in the downstream task, recent studies of SSL have made efforts in either upstream pre-training or downstream transfer (Zhang et al., 2021).”
• How Well Self-Supervised Pre-Training Performs with Streaming Data? (2021, methodology): “Most existing methods for fine-tuning [29, 30] are devised for supervised pre-trained models, while a recent study [63] starts to pay attention to the fine-tuning of self-supervised pre-trained models.”