Adversarial Network Compression

By Vasileios Belagiannis, Azade Farshad, Fabio Galasso. 2018

61 citations recorded by the source corpus.

Neural network compression has recently received much attention due to the computational requirements of modern deep models. In this work, our objective is to transfer knowledge from a deep and accurate model to a smaller one. Our contributions are threefold: (i) we propose an adversarial network compression approach to train the small student network to mimic the large teacher, without the need for labels during training; (ii) we introduce a regularization scheme to prevent a trivially-strong discriminator without reducing the network capacity and (iii) our approach generalizes on different teacher-student models. In an extensive evaluation on five standard datasets, we show that our student has small accuracy drop, achieves better performance than other knowledge transfer approaches and it surpasses the performance of the same network trained with labels. In addition, we demonstrate state-of-the-art results compared to other compression strategies.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Variational Information Distillation for Knowledge Transfer (2019, background): “Recent works [3, 6, 13, 30, 31] relax the regularization of matching the entire layer by matching carefully designed features/statistics extracted from intermediate layers of the teacher and the student networks, e.”
Knowledge Distillation: A Survey (2020, methodology): “s introduced to distinguish the samples from the student and the teacher models by using either the logits (Xu et al., 2018a,b) or the features (Wang et al., 2018e), shown in Fig. 8 (b). Speciﬁcally, Belagiannis et al. (2018) used unlabeled data samples to form the knowledge transfer. Multiple discr”
CHAOS Challenge - Combined (CT-MR) Healthy Abdominal Organ Segmentation (2020, background): “, 2019), optimizing neural network architecture to reduce computational cost (Belagiannis et al., 2019; Carreira-Perpiñán and Idelbayev, 2018), attaching importance to repeatability and reproducibility (Nikolov et al.”
Knowledge Distillation for End-to-End Person Search (2019, unknown): “It has gained attention for its promise of more effective training [45], better accuracy [3] and efficiency [2, 41, 49, 58], but it remains strongly limited to networks solving the single classification task.”
Cross-Resolution Face Recognition via Prior-Aided Face Hallucination and Residual Knowledge Distillation (2019, methodology): “Belagiannis et al [2] apply adversarial strategy to knowledge distillation.”
Residual Knowledge Distillation (2020, background): “Belagiannis et al. [3] involved adversarial learning into KD by employing a discriminator to tell whether the outputs of S and T are close enough, Ashok et al. [1] exploited reinforcement learning to ﬁnd out the best network structure of S under the guidance of T , and Wang et al. [19], Gao et al.…”
Knowledge Squeezed Adversarial Network Compression (2019, methodology): “We adjust the teacher network to ResNet-152 like ANC [14] and TSCAN [15] since the pretrain model is available, the student network is changed to ResNet-50/18, the mini-batch size is set to 128, and the total epoch is altered to 120 for ImageNet dataset, where the learning rate is divided by 10 at e”
Training Student Networks for Acceleration with Conditional Adversarial Networks (2018, background): “A recent preprint [3] appears a few months later than ours has a similar approach for network compression.”
Compressing GANs using Knowledge Distillation (2019, unknown): “There has been recent progress in the area of neural network compression (Liu et al., 2019) (Ba & Caurana, 2013) (Urban et al., 2017) and various compression techniques have been exercised to solve this problem (Belagiannis et al., 2018) (Xu et al., 2017) (Yim et al., 2017) (Kim & Kim, 2017).”
Complementary Relation Contrastive Distillation (2021, background): “Knowledge distillation aims to transfer the knowledge from one deep learning model (the teacher) to another (the student), such as distilling a large network into a smaller one [19, 49, 2, 48, 12] or ensembling a collection of models into a single model [29, 37, 27, 45].”