Few Shot Network Compression via Cross Distillation

By Haoli Bai, Jiaxiang Wu, Irwin King, Michael R. Lyu. 2019

69 citations recorded by the source corpus.

Model compression has been widely adopted to obtain light-weighted deep neural networks. Most prevalent methods, however, require fine-tuning with sufficient training data to ensure accuracy, which could be challenged by privacy and security issues. As a compromise between privacy and performance, in this paper we investigate few shot network compression: given few samples per class, how can we effectively compress the network with negligible performance drop? The core challenge of few shot network compression lies in high estimation errors from the original network during inference, since the compressed network can easily over-fits on the few training instances. The estimation errors could propagate and accumulate layer-wisely and finally deteriorate the network output. To address the problem, we propose cross distillation, a novel layer-wise knowledge distillation approach. By interweaving hidden layers of teacher and student network, layer-wisely accumulated estimation errors can be effectively reduced. The proposed method offers a general framework compatible with prevalent network compression techniques such as pruning. Extensive experiments n benchmark datasets demonstrate that cross distillation can significantly improve the student network's accuracy when only a few training instances are available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qun Liu
2. Xin Jiang
3. Dacheng Tao
4. Wei Zhang
5. Lifeng Shang
6. Lu Hou
7. Lin Wang
8. Kuk-Jin Yoon

In Computer Vision and Pattern Recognition

1. Wei Zhang
2. Qun Liu
3. Dacheng Tao
4. Michael R. Lyu
5. Irwin King
6. Lin Wang
7. Kuk-Jin Yoon
8. Xin Jiang

Citations in the corpus

• BinaryBERT: Pushing the Limit of BERT Quantization (2020, background): “, 2018), improving training objectives (Martinez et al., 2020; Bai et al., 2020) and reduction of gradient mismatch (Bai et al.”
• Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “pruning to achieve few-shot compression which requires few data [298], [299], [300].”
• Progressive Network Grafting for Few-Shot Knowledge Distillation (2020, unknown): “The work of (Bai et al. 2019) designs a cross distillation method, combined with network pruning to reduce the layer-wise accumulated errors in the few-shot setting.”
• Deep Streaming Label Learning (2020, methodology): “Due to its effectiveness, KD has been being used in many fields such as object detection (Chen et al., 2017), model compression (Heo et al., 2019), image classification (Liu et al., 2018), machine translation (Zhou et al., 2020), and meta-learning (Bai et al., 2020).”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “[17] add a 1×1 layer after each pruned layer block in the student and estimate the least-squared error to align the parameters with the student.”