Dream Distillation: A Data-Independent Model Compression Framework
By Kartikeya Bhardwaj, Naveen Suda, Radu Marculescu. 2019
55 citations recorded by the source corpus.
Model compression is eminently suited for deploying deep learning on IoT-devices. However, existing model compression techniques rely on access to the original or some alternate dataset. In this paper, we address the model compression problem when no real data is available, e.g., when data is private. To this end, we propose Dream Distillation, a data-independent model compression framework. Our experiments show that Dream Distillation can achieve 88.5% accuracy on the CIFAR-10 test set without actually training on the original data!
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Daniel Soudry
- Elad Hoffer
- Arun Mallya
- Tinne Tuytelaars
- José Manuel Álvarez
- Joost van de Weijer
- Hongxu Yin
- Itay Hubara
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jitendra Malik
- Ming-Hsuan Yang
- Xiaogang Wang
- Yi Yang
- Bernt Schiele
Citations in the corpus
- Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, background): “Along the same line, Bhardwaj et al . explored the training data cluster centroids to improve inversion efficacy [3].”
- The Knowledge Within: Methods for Data-Free Model Compression (2019, methodology): “In contrast to previous attempts [17, 3] which required sampling activation generated by real data, our method uses the recorded knowledge on low order statistics captured by BN layers (i.”
- Few Shot Network Compression via Cross Distillation (2019, methodology): “To take care of security issues in network compression, some recent works (Chen et al. 2019; Bhardwaj, Suda, and Marculescu 2019; Lopes, Fenu, and Starner 2017) motivate from knowledge distillation (Hinton, Vinyals, and Dean 2015; Romero et al. 2014), and propose data-free fine-tuning by…”
- Data-free Knowledge Distillation for Object Detection (2021, methodology): “Additionally, unlike previous data-free distillation approaches that only work on image classification models [2, 22, 28], we distill deep object detector networks.”
- Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation (2020, background): “[15, 17, 2, 1]) compose synthetic transfer set and achieve effective distillation.”
- Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “[169] argue that the metadata used in [21], [143] are actually not complete data-free approaches since the metadata is formed using training data itself.”
- Half-Real Half-Fake Distillation for Class-Incremental Semantic Segmentation (2021, unknown): “For fake data generation, we use Adam optimizer as in [2] with learning rate 0.25.”