Adaptive Neural Networks for Efficient Inference

By Tolga Bolukbasi, Joseph Wang, Ofer Dekel, Venkatesh Saligrama. 2017

425 citations recorded by the source corpus.

We present an approach to adaptively utilize deep neural networks in order to reduce the evaluation time on new examples without loss of accuracy. Rather than attempting to redesign or approximate existing networks, we propose two schemes that adaptively utilize networks. We first pose an adaptive network evaluation scheme, where we learn a system to adaptively choose the components of a deep network to be evaluated for each example. By allowing examples correctly classified using early layers of the system to exit, we avoid the computational time associated with full evaluation of the network. We extend this to learn a network selection system that adaptively selects the network to be evaluated for each example. We show that computational time can be dramatically reduced by exploiting the fact that many examples can be correctly classified using relatively efficient networks and that complex, computationally costly networks are only necessary for a small fraction of examples. We pose a global objective for learning an adaptive early exit or network selection policy and solve it by reducing the policy learning problem to a layer-by-layer weighted binary classification problem. Empirically, these approaches yield dramatic reductions in computational cost, with up to a 2.8x speedup on state-of-the-art networks from the ImageNet image recognition challenge with minimal (<1%) loss of top5 accuracy.

Subfields: Artificial Intelligence, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “early exiting is typically achieved according to confidencebased criteria [45], [51] or learned functions [46], [52], [53].”
Why Should We Add Early Exits to Neural Networks? (2020, methodology): “[ 72, 55, 12, 32 ] have considered a variation of LT, in which the backbone network is trained separately from each of the auxiliary classiﬁer.”
The Right Tool for the Job: Matching Model and Instance Complexities (2020, background): “Bolukbasi et al. (2017) and Huang et al.”
Depth-Adaptive Transformer (2019, background): “(2017) and Bolukbasi et al. (2017) to structured prediction where dynamic computation decisions impact future computation.”
Federated Learning: Challenges, Methods, and Future Directions (2019, background): “Similar comprehensive analyses have been performed for efficient neural network inference [3] and are necessary to compare communication-reduction techniques for federated learning in a meaningful way.”
NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection (2019, background): “Such early exit idea is related to [3, 37], especially in image classification [14].”
SkipNet: Learning Dynamic Routing in Convolutional Networks (2017, background): “Related to the work on early termination, several others [1, 9, 21, 33] investigate cascaded model composition.”
Measuring and Narrowing the Compositionality Gap in Language Models (2022, background): “…problems has been previously explored in both language (Graves, 2016; Seo et al., 2017; Dong et al., 2019; Schwartz et al., 2020; Schwarzschild et al., 2021; Nye et al., 2021; Wei et al., 2022b; Zhou et al., 2022) and vision (Bolukbasi et al., 2017; Huang et al., 2018; Wang et al., 2018, 2021).”
Designing Adaptive Neural Networks for Energy-Constrained Image Classification (2018, methodology): “outperforms layer-level execution [1].”
PonderNet: Learning to Ponder (2021, background): “Another approach is represented by Adaptive Early Exit Networks (Bolukbasi et al., 2017) where the forward pass of an existing network is terminated at evaluation time if it is likely that the part of the network used so far already predicts the correct answer.”
Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference (2019, background): “Another approach is to use early-exit branches in the network [7, 35, 45].”
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing (2019, background): “[89] train a policy that determines whether the current samples should proceed to the next layer by adding regularization to the evaluation latency of the DNN model.”
Energy-efficient Amortized Inference with Cascaded Deep Classifiers (2017, background): “Adaptive Neural Network [Bolukbasi et al. , 2017](ANN) is the most relevant approach to ours.”
Surprisal-Triggered Conditional Computation with Neural Networks (2020, methodology): “To avoid the difficulties of reinforcement learning, often a soft approximation to the hard selection decision is used during training [11, 35, 36, 37, 38, 39, 40, 41, 42].”
Pay Attention to Features, Transfer Learn Faster CNNs (2020, methodology): “There are methods that dynamically select which paths to evaluate in a network dependent on the input (Figurnov et al., 2017; Dong et al., 2017b; Bolukbasi et al., 2017; Lin et al., 2017; Shazeer et al., 2017; Wu et al., 2018; Ren et al., 2018).”
ILFO: Adversarial Attack on Adaptive Neural Networks (2020, background): “Bolukbasi et al.[2] train a termination policy for cascades of pre-trained CNNs arranged in order of increasing costs.”
Dynamic Channel Pruning: Feature Boosting and Suppression (2018, background): “A common approach to overcome this is to solve (He et al., 2017) or learn (Liu et al., 2017; Ye et al., 2018) channel saliencies to drive the sparsiﬁcation of CNNs.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “Going deeper with conditional computing abandoning RL, Bolukbasi [25] reports a system that contains a intact combination of the current SOTA networks (e.”
Dynamic-stride-net: deep convolutional neural network with dynamic stride (2019, background): “Moreover, network acceleration can be improved using cascaded models of various pretrained deep convolutional neural networks [15,16,17,18].”