SEED: Self-supervised Distillation For Visual Representation

By Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, Zicheng Liu. 2021

217 citations recorded by the source corpus.

This paper is concerned with self-supervised learning for small models. The problem is motivated by our empirical studies that while the widely used contrastive self-supervised learning method has shown great progress on large model training, it does not work well for small models. To address this problem, we propose a new learning paradigm, named SElf-SupErvised Distillation (SEED), where we leverage a larger network (as Teacher) to transfer its representational knowledge into a smaller architecture (as Student) in a self-supervised fashion. Instead of directly learning from unlabeled data, we train a student encoder to mimic the similarity score distribution inferred by a teacher over a set of instances. We show that SEED dramatically boosts the performance of small networks on downstream tasks. Compared with self-supervised baselines, SEED improves the top-1 accuracy from 42.2% to 67.6% on EfficientNet-B0 and from 36.3% to 68.2% on MobileNet-v3-Large on the ImageNet-1k dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Emerging Properties in Self-Supervised Vision Transformers (2021, background): “Previous works have also combined self-supervised learning and knowledge distillation [25, 63, 13, 47], enabling self-supervised model compression and performance gains.”
Compressing Visual-linguistic Model via Knowledge Distillation (2021, methodology): “[17, 62] propose using a sample queue to store history embeddings and show that contrasting with more negative samples is beneficial for knowledge distillation.”
Continual Contrastive Self-supervised Learning for Image Classification (2021, background): “The learned representations in the small network by SEED[17] can be used for multiple downstream tasks.”
Seed the Views: Hierarchical Semantic Alignment for Contrastive Representation Learning (2022, methodology): “As shown in Table 3, when apply CSML to ResNet-18 and EfﬁcientNet-B0 [58], we improve the MoCo v2 baseline (we adopt the results of ResNet18 and EfﬁcientNet-B0 reported in [54] as our baseline) by 5.2% and 19.8% respectively, which validate the effectiveness of CSML on different backbones.”
S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration (2021, methodology): “A concurrent study SEED [11] also employed self-supervised distillation loss, which can be considered as a contemporaneous work of ours.”
Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation (2021, methodology): “Inspired by [13], we develop a mutual knowledge distillation method between the local and global branches for novel category discovery without using labels.”
PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining (2022, methodology): “Downstream Object Detection Following [33, 34], for the downstream object detection task, we fine tune the whole model.”
Online Continual Learning for Embedded Devices (2022, background): “, ResNet-50) and perform poorly when applied directly to mobile architectures (Fang et al., 2021; Abbasi Koohpayegani et al., 2020).”
Online Prototype Learning for Online Continual Learning (2023, methodology): “In addition, at each iteration, PCL needs to cluster all samples M times, which is very expensive for training, while OPE only needs to compute online prototypes once.”
R-Drop: Regularized Dropout for Neural Networks (2021, background): “Minimizing the KL-divergence between the output distributions of two different models correlates with knowledge distillation [23, 15, 2, 36, 13, 78], where the two models refer to teacher and student, respectively.”
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2021, background): “SEED distillation proposed by (Fang et al., 2021) has a similar spirit as the contrastive distillation.”
ISD: Self-Supervised Learning by Iterative Similarity Distillation (2020, methodology): “Also, our method benefits from different temperatures for the teacher and student networks which is inspired by [14].”
A Fast Knowledge Distillation Framework for Visual Recognition (2021, methodology): “Knowledge Distillation (KD) [16] has been a widely used technique in various visual domains, such as the supervised recognition [2, 24, 30, 35, 47, 48] and self-supervised representation learning [4, 10, 33].”