Online Ensemble Model Compression Using Knowledge Distillation

By Devesh Walawalkar, Zhiqiang Shen, Marios Savvides. 2020

59 citations recorded by the source corpus.

This paper presents a novel knowledge distillation based model compression framework consisting of a student ensemble. It enables distillation of simultaneously learnt ensemble knowledge onto each of the compressed student models. Each model learns unique representations from the data distribution due to its distinct architecture. This helps the ensemble generalize better by combining every model's knowledge. The distilled students and ensemble teacher are trained simultaneously without requiring any pretrained weights. Moreover, our proposed method can deliver multi-compressed students with single training, which is efficient and flexible for different scenarios. We provide comprehensive experiments using state-of-the-art classification models to validate our framework's effectiveness. Notably, using our framework a 97% compressed ResNet110 student model managed to produce a 10.64% relative accuracy gain over its individual baseline training on CIFAR100 dataset. Similarly a 95% compressed DenseNet-BC(k=12) model managed a 8.17% relative accuracy gain.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Baosheng Yu
5. Lan Du
6. Liyuan Sun
7. Gabriella Csurka
8. Jing Zhang

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Baosheng Yu
5. Lan Du
6. Liyuan Sun
7. Luc Van Gool
8. Trevor Darrell

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “A variety of online knowledge distillation methods have been proposed, especially in the last three years (Zhang et al., 2018b; Chen et al., 2020a; Xie et al., 2019; Anil et al., 2018; Kim et al., 2019b; Zhou et al., 2018; Walawalkar et al., 2020).”
• MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks (2020, unknown): “Some works also proposed online distillation [51, 60] that do not rely on a pre-trained teacher, so teacher and student can be learned simultaneously.”
• Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, background): “In contrast, online distillation provides a one-phase end-to-end training scheme via teacher-student collaborative learning on a peer-network architecture instead of a fixed one [25, 33, 36, 32, 28, 12].”