Boosting Self-Supervised Learning via Knowledge Transfer

By Mehdi Noroozi, Ananth Vinjimoor, Paolo Favaro, Hamed Pirsiavash. 2018

320 citations recorded by the source corpus.

In self-supervised learning, one trains a model to solve a so-called pretext task on a dataset without the need for human annotation. The main objective, however, is to transfer this model to a target domain and task. Currently, the most effective transfer strategy is fine-tuning, which restricts one to use the same model or parts thereof for both pretext and target tasks. In this paper, we present a novel framework for self-supervised learning that overcomes limitations in designing and comparing different tasks, models, and data domains. In particular, our framework decouples the structure of the self-supervised model from the final task-specific fine-tuned model. This allows us to: 1) quantitatively assess previously incompatible models including handcrafted features; 2) show that deeper neural network models can learn better representations from the same pretext task; 3) transfer knowledge learned with a deep model to a shallower one and thus boost its learning. We use this framework to design a novel self-supervised task, which achieves state-of-the-art performance on the common benchmarks in PASCAL VOC 2007, ILSVRC12 and Places by a significant margin. Our learned features shrink the mAP gap between models trained via self-supervised learning and supervised learning from 5.9% to 2.6% in object detection on PASCAL VOC 2007.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

S4L: Self-Supervised Semi-Supervised Learning (2019, background): “Follow-up papers [24, 26] generalize this idea for predicting a permutation of multiple randomly sampled and permuted patches.”
Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “Previous works have also combined self-supervised learning and knowledge distillation, enabling self-supervised model compression [20] and performance gains [12, 38].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “Prior pretext tasks include image colorization [9, 30, 38, 39, 86, 87], orientation prediction [20], affine transform prediction [85], predicting contextual image patches [10], reordering image patches [5, 21, 53, 54, 56], counting visual primitives [55], or their combinations [11].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, background): “…various cues and tasks namely, in-painting (Pathak et al., 2016), patch context and jigsaw puzzles (Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2018; Mundhenk et al., 2017), clustering (Caron et al., 2018), noise-as-targets (Bojanowski & Joulin, 2017), colorization (Zhang et…”
Revisiting Self-Supervised Visual Representation Learning (2019, methodology): “These include a model from [31] that predicts the permutation of a “jigsaw puzzle” created from the full image and recent follow-ups [29, 33].”
Self-labelling via simultaneous clustering and representation learning (2019, background): “In (Noroozi et al., 2018) a large network is trained with a pretext task and a smaller network is trained via knowledge transfer of the clustered data.”
Knowledge Distillation: A Survey (2020, background): “…domains (Asami et al. 2017), catastrophic forgetting (Lee et al. 2019b), accelerating learning of deep models (Chen et al. 2016), efﬁciency of neural architecture search (Bashivan et al. 2019), self-supervision (Noroozi et al. 2018), and data augmentation (Lee et al. 2019a; Gordon and Duh 2019).”
Self-Supervised Learning: Generative or Contrastive (2020, methodology): “methods [13], [73], [87], [137], especially the DeepCluster [13] which first achieves competitive performance to the supervised model AlexNet [67].”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, background): “…such as the boundary of the objects [28, 38], the context of images [25, 36], and the similarity of two patches from an image [6, 15, 33, 34, 48], and the other is the temporal based methods, which exploit temporal connection between frames such as the temporal order of frames [10, 26,…”
Cross and Learn: Cross-Modal Self-Supervision (2018, background): “As for static images, there are several approaches exploiting color information [42,20] and spatial context [6,26,28].”
Cross Pixel Optical Flow Similarity for Self-Supervised Learning (2018, neutral): “e the in-painting problem, where a network is tasked with lling-in partially deleted parts of an image. [22] predict the relative position of two patches extracted from an image. In a similar spirit, [28,29] solve a jigsaw puzzle problem. [29] also cluster features from a pre-trained network to gene”
Are we pretraining it right? Digging deeper into visio-linguistic pretraining (2020, background): “This work spawned multiple works around predicting the “jigsaw puzzle” [35,36].”
Self-Supervised CycleGAN for Object-Preserving Image-to-Image Domain Adaptation (2020, methodology): “Various proxy tasks have been proposed, which include grayscale image colorization [15], jigsaw puzzles [23] and object motion estimation [16].”
A Cookbook of Self-Supervised Learning (2023, methodology): “This strategy was superseded by “jigsaw” methods [Pathak et al., 2016, Noroozi et al., 2018] that break an image into an array of disjoint patches and predict the relative location of each.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Using pseudo labels is the most common approach, as done in [119], [170].”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, methodology): “Then a ConvNet is trained to recognize the cluster assignment [34], [44] or to recognize whether two imaged are from same cluster [43].”
Knowledge Distillation Meets Self-Supervision (2020, methodology): “Based on pretext tasks, SS methods can be grouped into several categories, including constructionbased methods such as inpainting [35] and colorization [50], prediction-based methods [7,9,11,21,28,30,31,47,49], cluster-based methods [3,48], generation-based methods [8,10,12] and contrastive-based me”
ISD: Self-Supervised Learning by Iterative Similarity Distillation (2020, unknown): “Techniques from knowledge distillation can also be used in an unsupervised way to improve selfsupervised learning [1, 31, 46].”
Charting the Right Manifold: Manifold Mixup for Few-shot Learning (2019, methodology): “Motivated by the task of context prediction, the pretext task was extended to predict the permutation of the shuffled image patches [41, 39, 43].”