SkipNet: Learning Dynamic Routing in Convolutional Networks

By Xin Wang, Fisher Yu, Zi-Yi Dou, Joseph E. Gonzalez. 2017

763 citations recorded by the source corpus.

While deeper convolutional networks are needed to achieve maximum accuracy in visual perception tasks, for many inputs shallower networks are sufficient. We exploit this observation by learning to skip convolutional layers on a per-input basis. We introduce SkipNet, a modified residual network, that uses a gating network to selectively skip convolutional blocks based on the activations of the previous layer. We formulate the dynamic skipping problem in the context of sequential decision making and propose a hybrid learning algorithm that combines supervised learning and reinforcement learning to address the challenges of non-differentiable skipping decisions. We show SkipNet reduces computation by 30-90% while preserving the accuracy of the original model on four benchmark datasets and outperforms the state-of-the-art dynamic networks and static compression methods. We also qualitatively evaluate the gating policy to reveal a relationship between image scale and saliency and the number of layers skipped.

Subfields: Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “Our work is also related to the conditional computation models that activate parts of the neural network based on the input [2, 1, 33, 12, 36].”
Consistent Accelerated Inference via Confident Adaptive Transformers (2021, background): “[41] Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E.”
Depth-Adaptive Transformer (2019, methodology): “al. (2017) to structured prediction where dynamic computation decisions impact future computation. Related work from computer vision includes Teerapittayanon et al. (2016); Figurnov et al. (2017) and Wang et al. (2018) who explored the idea of dynamic routing either by exiting early or by skipping l”
Once for All: Train One Network and Specialize it for Efficient Deployment (2019, background): “For example, [17, 26, 28] propose to learn an additional controller or gating modules to adaptively drop blocks in a given neural network; [13] introduces early-exit branches in the computation graph, allowing to exit in the middle based on the current prediction confidence; [15] proposes to adaptiv”
Conditional Channel Gated Networks for Task-Aware Continual Learning (2020, background): “Such gates may either drop entire residual blocks [36, 37] or specific units within a layer [6, 3].”
Slimmable Neural Networks (2018, background): “Wu et al. (2017) and Wang et al. (2017) propose to selectively choose the blocks in a deep residual network to execute during inference.”
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (2019, background): “further extended the layer-wise dropout from training to inference by introducing additional controller modules or gating functions based on the current input [37, 36].”
Measuring and Narrowing the Compositionality Gap in Language Models (2022, background): “…problems has been previously explored in both language (Graves, 2016; Seo et al., 2017; Dong et al., 2019; Schwartz et al., 2020; Schwarzschild et al., 2021; Nye et al., 2021; Wei et al., 2022b; Zhou et al., 2022) and vision (Bolukbasi et al., 2017; Huang et al., 2018; Wang et al., 2018, 2021).”
BERT Loses Patience: Fast and Robust Inference with Early Exit (2020, methodology): “In addition, existing approaches [24, 28] trained separate models to determine passing through or skipping each layer.”
CondConv: Conditionally Parameterized Convolutions for Efficient Inference (2019, background): “BlockDrop [47] and SkipNet [45] use reinforcement learning to learn the subset of blocks needed to process a given input.”
Network Slimming by Slimmable Networks: Towards One-Shot Architecture Search for Channel Numbers (2019, background): “The most common constraints [1, 2, 3, 4, 5], i.e ., latency, FLOPs and runtime memory footprint, are all bound to the number of channels.”
Else-Net: Elastic Semantic Network for Continual Action Recognition from Skeleton Data (2021, background): “[33] designed a dynamic network, called SkipNet, that adaptively adjusts the network architecture by selecting or skipping convolutional layers based on the input data.”
ILFO: Adversarial Attack on Adaptive Neural Networks (2020, methodology): “The two AdNNs we have attacked are SkipNet [29] and SACT [7].”
Dynamic Routing Networks (2019, methodology): “Compared with SkipNet and AIG, the hypernetworks of DRNets are optimized more effectively with a two-stage temperature annealing scheme, and DRNets support a varied branch selection of multiple branches for each connection instead of the binary skipping choice.”