Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution

By Lanlan Liu, Jia Deng. 2017

216 citations recorded by the source corpus.

We introduce Dynamic Deep Neural Networks (D2NN), a new type of feed-forward deep neural network that allows selective execution. Given an input, only a subset of D2NN neurons are executed, and the particular subset is determined by the D2NN itself. By pruning unnecessary computation depending on input, D2NNs provide a way to improve computational efficiency. To achieve dynamic selective execution, a D2NN augments a feed-forward deep neural network (directed acyclic graph of differentiable modules) with controller modules. Each controller module is a sub-network whose output is a decision that controls whether other modules can execute. A D2NN is trained end to end. Both regular and controller modules in a D2NN are learnable and are jointly trained to optimize both accuracy and efficiency. Such training is achieved by integrating backpropagation with reinforcement learning. With extensive experiments of various D2NN architectures on image classification tasks, we demonstrate that D2NNs are general and flexible, and can effectively optimize accuracy-efficiency trade-offs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “The simplest dynamic routing can be implemented by selectively executing one of multiple candidate modules at each layer [96], [97], which is equivalent to producing a one-hot probability distribution α s→· in Eq.”
Once for All: Train One Network and Specialize it for Efficient Deployment (2019, background): “For example, [17, 26, 28] propose to learn an additional controller or gating modules to adaptively drop blocks in a given neural network; [13] introduces early-exit branches in the computation graph, allowing to exit in the middle based on the current prediction confidence; [15] proposes to adaptiv”
Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning (2017, methodology): “The best performer is the WPL algorithm which outperforms the nearest competitor, tabular Q-Learning by about 4%.”
Slimmable Neural Networks (2018, background): “Liu & Deng (2017) introduce controller modules whose outputs control whether to execute other modules.”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, background): “These gating functions can be learned with reinforcement learning [2, 34, 11].”
SpotTune: Transfer Learning Through Adaptive Fine-Tuning (2018, methodology): “Our proposed approach is related to conditional computation methods [4, 27, 11], which aim to dynamically route information in neural networks with the goal of improving computational efficiency.”
Interpretable Image Recognition with Hierarchical Prototypes (2019, background): “A few identify subsections of an image that were important to a classification, e.g., the class-attention maps of Pinheiro and Collobert (2015) and Zhou et al. (2016).”
DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution (2020, background): “Conditional Convolution Conditional convolutional networks adopt dynamic kernels, widths, or depths, e.g ., [16,39,43,48,74,77].”
AutoFocus: Efficient Multi-Scale Inference (2018, background): “While there is work on image classification which performs conditional computation [4, 39, 68], modern object detection algorithms perform static inference and process every pixel of a multi-scale image pyramid to detect objects of different sizes [60, 61, 54].”
Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks (2017, methodology): “The proposed model can be encouraged to perform fewer state updates through additional loss terms, a common practice in neural networks with dynamically allocated computation [16, 17, 6, 10].”
S2DNAS: Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search (2019, background): “Recently, dynamic inference has emerged as a promising alternative to speed up the CNN inference by dynamically changing the computation graph according to each input sample [39, 2, 6, 5, 45, 16, 28, 7].”
LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition (2019, background): “Recently, the idea of conditional computation has also been investigated in deep neural networks [30, 15, 24, 6, 1, 22] through learning when to exit CNNs with attached decision branches.”
Dynamic ReLU (2020, methodology): “Dynamic Neural Networks: Our method is related to recent work of dynamic neural networks [20,23,34,37,42,15,14,41,3].”
Dynamic Convolution: Attention Over Convolution Kernels (2019, methodology): “Dynamic Deep Neural Networks: Our method is related to recent works of dynamic neural networks [17, 20, 29, 32, 37, 13] that focus on skipping part of an existing model based on input image.”
HydraNets: Specialized Dynamic Architectures for Efficient Inference (2018, background): “Recent work [22, 23] has explored directly incorporating inference cost in the optimization and explore training methods for jointly learning the routing and the network features.”
InstaNAS: Instance-aware Neural Architecture Search (2018, methodology): “Several conditional computation methods have been proposed to dynamically execute different modules of a model on a per-example basis [2, 12, 15, 25, 28, 27].”