Deciding How to Decide: Dynamic Routing in Artificial Neural Networks

By Mason McGill, Pietro Perona. 2017

109 citations recorded by the source corpus.

We propose and systematically evaluate three strategies for training dynamically-routed artificial neural networks: graphs of learned transformations through which different input signals may take different paths. Though some approaches have advantages over others, the resulting networks are often qualitatively similar. We find that, in dynamically-routed networks trained to classify images, layers and branches become specialized to process distinct categories of images. Additionally, given a fixed computational budget, dynamically-routed networks tend to perform better than comparable statically-routed networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “Based on the multi-scale architecture design, researchers have also studied the exiting policies [54], [55] (see Fig.”
Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning (2017, methodology): “These include decisions trained with REINFORCE (Denoyer & Gallinari, 2014), (Bengio et al., 2015), (Hamrick et al., 2017), Q Learning (Liu & Deng, 2017), and actor-critic methods (McGill & Perona, 2017).”
SkipNet: Learning Dynamic Routing in Convolutional Networks (2017, methodology): “Close to our proposed algorithm, [9] and [21] study reinforcement learning algorithms to solve the decision problem.”
Multigrid Neural Architectures (2016, background): “Concurrent work extends such exploration to include cascades [14, 26], in which explicit routing decisions allow for partial evaluation of a network.”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “Dynamic computation to improve efﬁciency has been studied from multiple perspectives [1, 2, 50, 54, 15, 37, 13, 33].”
NBDT: Neural-Backed Decision Trees (2020, unknown): “In particular, several redesigned neural network architectures utilize decision tree branching structures [35,21,34].”
AdaFrame: Adaptive Frame Selection for Fast Video Recognition (2018, background): “Several recent approaches propose to add decision branches to different layers of CNNs to learn whether to exit the model [23, 9, 13, 4].”
Anytime Recognition with Routing Convolutional Networks (2019, background): “[28] proposes three novel learning targets and regularization methods to learn the routing policy in a neural network.”
HydraNets: Specialized Dynamic Architectures for Efficient Inference (2018, background): “Recent work [22, 23] has explored directly incorporating inference cost in the optimization and explore training methods for jointly learning the routing and the network features.”
InferLine: latency-aware provisioning and scaling for prediction serving pipelines (2020, methodology): “This conditional evaluation pattern appears in bandit algorithms [3, 20] used for model personalization as well as more general cascaded prediction pipelines [2, 14, 24, 34].”
Learning Instance-wise Sparsity for Accelerating Deep Models (2019, background): “[Teerapittayanon et al. , 2016; Liu and Deng, 2018; McGill and Perona, 2017; Teja Mullapudi et al. , 2018; Bolukbasi et al. , 2017] further proposed the branch selection operation to allow the learned neural networks to change themselves according to different input data.”
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition (2020, unknown): “Several works have been proposed that add decision branches to different layers of CNNs to learn whether to exit the network for faster inference [18,38].”
Learning Time/Memory-Efficient Deep Architectures with Budgeted Super Networks (2017, background): “Other approaches are dynamic networks which conditionally select the modules to respect a budget objective.[2, 22, 14, 1, 18].”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, methodology): “Many adaptive computation methods have been recently proposed with the goal of improving computational efficiency [4, 5, 50, 54, 18, 14, 33, 34].”
Channel Selection Using Gumbel Softmax (2020, background): “More recently, [19] and [34] both propose a cascading architecture which computes features at multiple scales and allows for dynamic evaluation, where at inference time the user can tradeoﬀ speed for accuracy.”
An end-to-end approach for speeding up neural network inference (2018, background): “More recently, [18] and [35] both propose a cascading architecture which computes features at multiple scales and allows for dynamic evaluation, where at inference time the user can tradeoff speed for accuracy.”
Learning Time-Efficient Deep Architectures with Budgeted Super Networks (2017, methodology): “In the same vein, the model proposed in [15] aims at learning routes in Super networks.”
Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks (2017, methodology): “The proposed model can be encouraged to perform fewer state updates through additional loss terms, a common practice in neural networks with dynamically allocated computation [16, 17, 6, 10].”