Knowledge Squeezed Adversarial Network Compression

By Changyong Shu, Peng Li, Yuan Xie, Yanyun Qu, Longquan Dai, Lizhuang Ma. 2019

12 citations recorded by the source corpus.

Deep network compression has been achieved notable progress via knowledge distillation, where a teacher-student learning manner is adopted by using predetermined loss. Recently, more focuses have been transferred to employ the adversarial training to minimize the discrepancy between distributions of output from two networks. However, they always emphasize on result-oriented learning while neglecting the scheme of process-oriented learning, leading to the loss of rich information contained in the whole network pipeline. Inspired by the assumption that, the small network can not perfectly mimic a large one due to the huge gap of network scale, we propose a knowledge transfer method, involving effective intermediate supervision, under the adversarial training framework to learn the student network. To achieve powerful but highly compact intermediate information representation, the squeezed knowledge is realized by task-driven attention mechanism. Then, the transferred knowledge from teacher network could accommodate the size of student network. As a result, the proposed method integrates merits from both process-oriented and result-oriented learning. Extensive experimental results on three typical benchmark datasets, i.e., CIFAR-10, CIFAR-100, and ImageNet, demonstrate that our method achieves highly superior performances against other state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Fei Wang
2. Chen Qian
3. Yuang Liu
4. Xiaojie Li
5. Lin Wang
6. Jun Wang
7. Jianlong Wu
8. Yue Liao

In Computer Vision and Pattern Recognition

1. Jianlong Wu
2. Fei Wang
3. Yue Liao
4. Chen Qian
5. Yuang Liu
6. Xiaojie Li
7. Lin Wang
8. Jun Wang

Citations in the corpus

• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Based on the definition of layer group [261], in which a group of layers have same spatial size, AT [260], FSP [254], Jacobian [202], MEAL [198], KSANC [27] and Kulkarni et al.”
• Source-Free Domain Adaptation for Semantic Segmentation (2021, background): “Since then, a variety of KD methods has been presented for model compression [28, 2, 50, 32], domain adaptation [52, 53], and multi-modal learning [51, 14, 10].”
• Local Correlation Consistency for Knowledge Distillation (2020, background): “[22] incorporate the intermediate supervision under the adversarial training framework.”