Data Distillation: Towards Omni-Supervised Learning

By Ilija Radosavovic, Piotr Doll'ar, Ross B. Girshick, Georgia Gkioxari, Kaiming He. 2017

447 citations recorded by the source corpus.

We investigate omni-supervised learning, a special regime of semi-supervised learning in which the learner exploits all available labeled data plus internet-scale sources of unlabeled data. Omni-supervised learning is lower-bounded by performance on existing labeled datasets, offering the potential to surpass state-of-the-art fully supervised methods. To exploit the omni-supervised setting, we propose data distillation, a method that ensembles predictions from multiple transformations of unlabeled data, using a single model, to automatically generate new training annotations. We argue that visual recognition models have recently become accurate enough that it is now possible to apply classic ideas about self-training to challenging real-world data. Our experimental results show that in the cases of human keypoint detection and general object detection, state-of-the-art models trained with data distillation surpass the performance of using labeled data from the COCO dataset alone.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Efficiency & Compression

Citations in the corpus

Self-Training With Noisy Student Improves ImageNet Classification (2019, background): “The main difference between Data Distillation and our method is that we use the noise to weaken the student, which is the opposite of their approach of strengthening the teacher by ensembling.”
Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation (2020, methodology): “We then show that the Copy-Paste augmentation strategy provides additional gains with self-training [44, 73] wherein we extract instances from ground-truth data and paste them onto unlabeled data annotated with pseudo-labels.”
Billion-scale semi-supervised learning for image classification (2019, methodology): “This strategy was shown successful for several tasks, like detection [34].”
Dataset Distillation (2018, methodology): “The main inspiration for this paper is network distillation (Hinton et al., 2015), a widely used technique in ensemble learning (Radosavovic et al., 2018) and model compression (Ba & Caruana, 2014; Romero et al., 2015; Howard et al., 2017).”
FEED: Feature-level Ensemble for Knowledge Distillation (2019, background): “This motivated researchers to develop many variants of it to various domains, and many researchers studied ways to better teach the student (Frosst and Hinton 2017; Pham et al. 2018; Radosavovic et al. 2018; Tan et al. 2018; Tarvainen and Valpola 2017).”
Similarity-Preserving Knowledge Distillation (2019, background): “As future work, we plan to explore similarity-preserving knowledge distillation in semisupervised and omni-supervised [29] learning settings.”
Knowledge Distillation: A Survey (2020, methodology): “Besides, there is a new type of distillation called data distillation, which is similar to data-free distillation (Radosavovic et al., 2018; Liu et al., 2019d; Zhang et al., 2020d).”
Rethinking Pre-training and Self-training (2020, background): “, [48–52]), who only study self-training in isolation or without a comparison against ImageNet pre-training.”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “In Data Distillation [40], additional annotation for object and key-point detection is obtained by ensembling and reﬁning a pretrained model’s prediction on unannotated images.”
Meta Pseudo Labels (2020, background): “On the other hand, label propagation losses typically enforce that the model is invariant against certain transformations of the data such as data augmentations, adversarial attacks, or proximity in the latent space [35, 64, 44, 5, 76, 30, 71, 58, 32, 51, 20].”
Relational Knowledge Distillation (2019, methodology): “[26] generate multiple predictions from an example by applying multiple data transformations on it, then use an ensemble of the predictions as annotations for omni-supervised learning.”
End-to-End Semi-Supervised Object Detection with Soft Teacher (2021, methodology): “In [20, 36], the predictions of different data augmentation are ensembled to form the pseudo labels for unlabeled images.”
MiniVLM: A Smaller and Faster Vision-Language Model (2020, methodology): “During VL pretraining, we apply data distillation [36] to add 7M Open Images [18] which are pseudo-labeled by the state-of-theart “teacher” captioning model.”
DistInit: Learning Video Representations Without a Single Labeled Video (2019, background): “Selecting confident predictions: As some recent work [39] has shown, distillation techniques can benefit from using only the most confident predictions for training the student.”
ST++: Make Self-trainingWork Better for Semi-supervised Semantic Segmentation (2021, methodology): “Recently, the traditional method is increasingly attracting attention from multiple fields, such as fully-supervised image recognition [35, 49, 48, 55], semi-supervised learning [43, 5, 15, 50] and domain adaptation [56, 57, 24].”
Teacher Supervises Students How to Learn From Partially Labeled Images for Facial Landmark Detection (2019, methodology): “(3) The specifically designed data augmentation [31, 42] is another direction to improve the accuracy and precision of the pseudo labels.”