Dual Dynamic Inference: Enabling More Efficient, Adaptive, and Controllable Deep Inference

By Yue Wang, Jianghao Shen, Ting-Kuei Hu, Pengfei Xu, Tan Minh Nguyen, Tan Y. Nguyen, Richard Baraniuk, Zhangyang Wang, Yingyan Lin. 2019

93 citations recorded by the source corpus.

State-of-the-art convolutional neural networks (CNNs) yield record-breaking predictive performance, yet at the cost of high-energy-consumption inference, that prohibits their widely deployments in resource-constrained Internet of Things (IoT) applications. We propose a dual dynamic inference (DDI) framework that highlights the following aspects: 1) we integrate both input-dependent and resource-dependent dynamic inference mechanisms under a unified framework in order to fit the varying IoT resource requirements in practice. DDI is able to both constantly suppress unnecessary costs for easy samples, and to halt inference for all samples to meet hard resource constraints enforced; 2) we propose a flexible multi-grained learning to skip (MGL2S) approach for input-dependent inference which allows simultaneous layer-wise and channel-wise skipping; 3) we extend DDI to complex CNN backbones such as DenseNet and show that DDI can be applied towards optimizing any specific resource goals including inference latency and energy cost. Extensive experiments demonstrate the superior inference accuracy-resource trade-off achieved by DDI, as well as the flexibility to control such a trade-off as compared to existing peer methods. Specifically, DDI can achieve up to 4 times computational savings with the same or even higher accuracy as compared to existing competitive baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Drawing early-bird tickets: Towards more efficient training of deep networks (2019, unknown): “…2017; Liu et al., 2018a), weight factorization (Denton et al., 2014), weight sharing (Wu et al., 2018a), quantization (Hubara et al., 2017), dynamic inference (Wang et al., 2018b; 2019b; Shen et al., 2020), network architecture search (Zoph & Le, 2017), among many others (Wang et al., 2018c;d).”
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges (2021, background): “Specifically, these studies [24, 67, 87, 109, 132, 140, 141, 148, 150, 155, 157, 158, 168, 172] define a loss function for each of the classifier, and minimize the weighted sum of cross entropy losses per sample as follows: where 3) by treating the final classifier (last exit) as teacher model and…”
Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations (2022, methodology): “There are also some attempts to accelerate the computation of the Transformer model, including proposing more efficient attention mechanisms (Wang et al., 2020a; Kitaev et al., 2020; Choromanski et al., 2020) and the compressed Transformer structures (Liu et al., 2021; Heo et al., 2021; Wang et…”
IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, methodology): “To overcame this issue, many adaptive computation methods are explored during the inference stage [1, 2, 57, 15, 26, 58, 20].”
Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference (2020, background): “A general multi-output network (Xu et al., 2019) could be decomposed by an ensemble of single-output models, with weight re-using enforced among them.”