BlockDrop: Dynamic Inference Paths in Residual Networks

By Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven J. Rennie, Larry S. Davis, Kristen Grauman, Rogério Schmidt Feris. 2017

530 citations recorded by the source corpus.

Very deep convolutional neural networks offer excellent recognition results, yet their computational expense limits their impact for many real-world applications. We introduce BlockDrop, an approach that learns to dynamically choose which layers of a deep network to execute during inference so as to best reduce total computation without degrading prediction accuracy. Exploiting the robustness of Residual Networks (ResNets) to layer dropping, our framework selects on-the-fly which residual blocks to evaluate for a given novel image. In particular, given a pretrained ResNet, we train a policy network in an associative reinforcement learning setting for the dual reward of utilizing a minimal number of blocks while preserving recognition accuracy. We conduct extensive experiments on CIFAR and ImageNet. The results provide strong quantitative and qualitative evidence that these learned policies not only accelerate inference but also encode meaningful visual information. Built upon a ResNet-101 model, our method achieves a speedup of 20% on average, going as high as 36% for some images, while maintaining the same 76.4% top-1 accuracy on ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Once for All: Train One Network and Specialize it for Efficient Deployment (2019, background): “, ResNet, MobileNetV2) [9, 28, 29], we have a much more diverse architecture space while allowing a significantly larger number of architectural configurations (10(19) v.”
MaskConnect: Connectivity Learning by Gradient Descent (2018, positive): “Finally, our approach is also related to conditional computation methods [34,35,36,37,38,39,40,41,42,43], which learn to drop out blocks of units.”
NISP: Pruning Networks Using Neuron Importance Score Propagation (2017, background): “[36] proposed to skip layers for speeding up inference.”
Slimmable Neural Networks (2018, background): “Wu et al. (2017) and Wang et al. (2017) propose to selectively choose the blocks in a deep residual network to execute during inference.”
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (2019, background): “further extended the layer-wise dropout from training to inference by introducing additional controller modules or gating functions based on the current input [37, 36].”
SpotTune: Transfer Learning Through Adaptive Fine-Tuning (2018, methodology): “Similar to [49], we generate all freeze/fine-tune decisions for all residual blocks at once, instead of relying on features of intermediate layers of the pre-trained model to obtain the fine-tuning policy.”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, background): “However, RL policy gradients are often complex, unwieldy to train and require techniques to reduce variance during training as well as it is slow to converge in many applications [58, 59, 23, 57].”
Channel Gating Neural Networks (2018, background): “Several recent efforts propose to use additional fully-connected layers [6, 8, 28] or recurrent networks [20, 29] to predict if a fraction of the computation can be skipped based on certain intermediate results produced by the CNN at run time.”
Mitigating Forgetting in Online Continual Learning via Instance-Aware Parameterization (2020, unknown): “specifically [3,1], [5,1], [3,2], [5,2], [3,8].”
Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference (2019, methodology): “Such practice is often called gating [11, 22], and can be applied at the level of convolutional layers [46, 48, 51], channels [3, 13, 30] or other elements in the network.”
Dynamic Modality Interaction Modeling for Image-Text Retrieval (2021, methodology): “Different from neural architecture search [20] with static inference, dynamic networks [17, 18, 28, 38] generate execution paths on the fly conditioned on input samples.”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “Representative meth-ods for image classiﬁcation, dynamically adjust network depth [13, 33, 59, 21, 63], width [66, 7, 20], perform routing [26, 33] or switch resolutions [62].”
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition (2020, unknown): “BlockDrop [60] effectively reduces the inference time by learning to dynamically select which layers to execute per sample during inference.”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, methodology): “BlockDrop [69] learns a policy to select the minimal number of blocks which are needed to correctly classify the given image by Markov decision process.”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, background): “ConvNet-AIG [45], SkipNet [48], and BlockDrop [50] exploit the robustness of Residual Networks (ResNets) to layer dropping and activate or deactivate full residual blocks conditioned on the input, to save compute.”
VA-RED2: Video Adaptive Redundancy Reduction (2021, methodology): “(Wu et al., 2018) develop a reinforcement learning approach to achieve this goal.”
Learning Dynamic Routing for Semantic Segmentation (2020, methodology): “Traditional methods mainly focus on image classification by dropping blocks [38, 17, 33, 36] or pruning channels [39, 20] for efficient inference.”