MEAL: Multi-Model Ensemble via Adversarial Learning

By Zhiqiang Shen, Zhankui He, Xiangyang Xue. 2018

157 citations recorded by the source corpus.

Often the best performing deep neural models are ensembles of multiple base-level networks. Unfortunately, the space required to store these many networks, and the time required to execute them at test-time, prohibits their use in applications where test sets are large (e.g., ImageNet). In this paper, we present a method for compressing large, complex trained ensembles into a single network, where knowledge from a variety of trained deep neural networks (DNNs) is distilled and transferred to a single DNN. In order to distill diverse knowledge from different trained (teacher) models, we propose to use adversarial-based learning strategy where we define a block-wise training loss to guide and optimize the predefined student network to recover the knowledge in teacher models, and to promote the discriminator network to distinguish teacher vs. student features simultaneously. The proposed ensemble method (MEAL) of transferring distilled knowledge with adversarial learning exhibits three important advantages: (1) the student network that learns the distilled knowledge with discriminators is optimized better than the original model; (2) fast inference is realized by a single forward pass, while the performance is even better than traditional ensembles from multi-original models; (3) the student network can learn the distilled knowledge from a teacher model that has arbitrary structures. Extensive experiments on CIFAR-10/100, SVHN and ImageNet datasets demonstrate the effectiveness of our MEAL method. On ImageNet, our ResNet-50 based MEAL achieves top-1/5 21.79%/5.99% val error, which outperforms the original model by 2.06%/1.14%.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, background): “Ensemble models on the other hand, have been shown to be promising teachers from the early work of [9] until recent works in various domains [10, 68, 54, 56] and with techniques to boost the their performance [52, 17, 41].”
Model Fusion via Optimal Transport (2019, neutral): “In a different line of work, Shen et al. [11] propose an approach where the student network is forced to produce outputs mimicking the teacher networks, by utilizing Generative Adversarial Network [12].”
Knowledge Distillation: A Survey (2020, methodology): “In particular, these KD-based speech recognition applications have spoken language identiﬁcation (Shen et al. 2018, 2019c, 2020), audio classiﬁcation (Gao et al. 2019; Perez et al. 2020), text-independent speaker recognition (Ng et al. 2018), speech enhancement (Watanabe et al. 2017), acoustic…”
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (2019, background): “Furthermore, some researchers extended knowledge distillation to generative adversarial problem [33, 27].”
Feature-map-level Online Adversarial Knowledge Distillation (2020, methodology): “, 2018) and MEAL (Shen et al., 2019) use the intermediate feature representation to transfer the”
An Adversarial Feature Distillation Method for Audio Classification (2019, methodology): “The knowledge distillation [23], [24] method is a teacher-student structure.”
Complementary Relation Contrastive Distillation (2021, background): “Knowledge distillation aims to transfer the knowledge from one deep learning model (the teacher) to another (the student), such as distilling a large network into a smaller one [19, 49, 2, 48, 12] or ensembling a collection of models into a single model [29, 37, 27, 45].”
MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks (2020, unknown): “[18] and our method.”
Dynamic Slimmable Network (2021, unknown): “As pointed out in [58, 59], an ensemble of teacher networks can generate more diverse, more accurate and more general soft labels for distillation training of the student network.”
ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions (2020, background): “There have been extensive studies on neural network compression and acceleration, including quantization [46,39,43], pruning [9,12,24,22], knowledge distillation [14,33,6] and compact network design [15,32,25,41].”
A Fast Knowledge Distillation Framework for Visual Recognition (2021, background): “Recently, many variants and extensions are proposed [6, 18, 24–26, 35, 37, 44, 47, 49, 54], such as employing internal feature representations [30], adversarial training with discriminators [32], transfer ﬂow [48], contrastive distillation [40], patient and consistent [2] etc.”
Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition (2020, methodology): “To further distill diverse knowledge, the proposed ensemble method (MEAL) (Shen, He, and Xue 2019) adopts an adversarial-based learning strategy.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “IR [67] and RKD [27] to regularize the context representation on the feature distributions of teacher and student. Some works also resort to the adversarial loss for measuring the quality of KD. MEAL [61] shows that the student learning the distilled knowledge with discriminators is better optimized”
Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study (2021, background): “…hard and soft labels with the cross-entropy loss for distillation, while our surprisingly good results and previous studies (Shen & Savvides, 2020; Shen et al., 2019; Bagherinezhad et al., 2018) indicate that knowledge distillation enabled by soft labels solely is not only an auxiliary…”
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, methodology): “To form a common objective for all paths, we can use a scheme analogous to ensemble distillation [59, 60] in supervised learning.”
Online Ensemble Model Compression Using Knowledge Distillation (2020, methodology): “Also in parallel, methods [22,34,1] were developed which focused on distilling not only the output knowledge but also the intermediate representations of teacher onto the student to have more effective knowledge transfer.”
Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning (2021, background): “For example, in [39] the teacher and the student are identical; in [38] the teacher is an ensemble of multiple networks; in [4, 9, 22] the teacher is a more complex network than the student for model compression; in [28] the teacher is a temporal ensemble of student checkpoints with the step of one ”
S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration (2021, background): “(3) which is equivalent to LKL following MEAL [31, 33].”