Improved Techniques for Training Adaptive Deep Networks

By Hao Li, Hong Zhang, Xiaojuan Qi, Ruigang Yang, Gao Huang. 2019

155 citations recorded by the source corpus.

Adaptive inference is a promising technique to improve the computational efficiency of deep models at test time. In contrast to static models which use the same computation graph for all instances, adaptive networks can dynamically adjust their structure conditioned on each input. While existing research on adaptive inference mainly focuses on designing more advanced architectures, this paper investigates how to train such networks more effectively. Specifically, we consider a typical adaptive deep network with multiple intermediate classifiers. We present three techniques to improve its training efficacy from two aspects: 1) a Gradient Equilibrium algorithm to resolve the conflict of learning of different classifiers; 2) an Inline Subnetwork Collaboration approach and a One-for-all Knowledge Distillation algorithm to enhance the collaboration among classifiers. On multiple datasets (CIFAR-10, CIFAR-100 and ImageNet), we show that the proposed approach consistently leads to further improved efficiency on top of state-of-the-art adaptive deep networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Shiji Song
2. Xiaodan Liang
3. Gao Huang
4. Xiaojun Chang
5. Yulin Wang
6. Rui Huang
7. Zuxuan Wu
8. Yu-Gang Jiang

In Computer Vision and Pattern Recognition

1. Xiaodan Liang
2. Luc Van Gool
3. Shiji Song
4. Gao Huang
5. Yi Yang
6. Dacheng Tao
7. Xiaogang Wang
8. Zuxuan Wu

In Optimization & Training

1. Shiji Song
2. Yulin Wang
3. Rui Huang
4. Le Yang
5. Mário Almeida
6. Gao Huang
7. Xin Dai
8. Tian Guo

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, methodology): “3 (b)) and training schemes [56] of early-exiting dynamic models.”
• Not All Images are Worth 16x16 Words: Dynamic Vision Transformers with Adaptive Sequence Length (2021, methodology): “In the context of recognition tasks, MSDNet and its variants [22, 49, 26] develop a multi-classifier CNN architecture to perform early exiting for easy samples.”
• EPNet: Learning to Exit with Flexible Multi-Branch Network (2020, methodology): “In the previous studies [10, 16, 20], the early-exiting policy is rule-based, e.g., if the uncertainty of the logit yield by f ia is below than a given threshold, the model directly early exits at the f i a [20].”
• Dynamic Slimmable Network (2021, methodology): “In previous dynamic networks [58, 59, 43, 62, 4, 27, 41, 26, 13, 9], the dynamic routing agent and the main network are jointly trained, analogous to jointly optimized NAS methods [45, 7, 61].”
• S2DNAS: Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search (2019, result): “Dynamic inference also refers to adaptive inference in previous works [41, 24].”
• SPINN: synergistic progressive inference of neural networks over device and cloud (2020, background): “This approach is further supported by the ML community’s growing number of works on progressive networks [23, 35, 48, 72, 81, 83, 84] which can be directly targeted by SPINN to yield an optimised deployment on mobile platforms.”
• Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges (2021, background): “Similar to the studies by Li et al. [77] and Phuong and Lampert [108], Liu et al. [84] leverage knowledge distillation [46] to train early classifiers, treating the final classifier of the BERT model and their introduced early classifiers as a teacher and student classifiers, respectively.”
• Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (2021, methodology): “In addition, IMTA [27] studies the training techniques of multi-exit models, which are actually complementary to DVT.”
• Resolution Adaptive Networks for Efficient Inference (2020, methodology): “As recent research in [22] investigates improved techniques for training adaptive networks, we further evaluate these techniques on both RANet and MSDNet.”
• AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, unknown): “3.3.”
• HMS: Hierarchical Modality Selection for Efficient Video Recognition (2021, methodology): “Conditional computation methods have been proposed for both image and video recognition to reduce computation while maintaining accuracy by dynamically performing early exiting in deep networks [2, 16], varying input resolutions [39, 49], skipping network layers or blocks [43] and selecting smaller ”
• Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification (2020, background): “For example, MSDNet [20] and its variants [32, 64] introduce a multi-scale architecture with multiple classifiers that enables it to adopt small networks for easy samples while switch to large models for hard ones.”