Spatially Adaptive Computation Time for Residual Networks

By Michael Figurnov, Maxwell D. Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry P. Vetrov, Ruslan Salakhutdinov. 2016

382 citations recorded by the source corpus.

This paper proposes a deep learning architecture based on Residual Network that dynamically adjusts the number of executed layers for the regions of the image. This architecture is end-to-end trainable, deterministic and problem-agnostic. It is therefore applicable without any modifications to a wide range of computer vision problems such as image classification, object detection and image segmentation. We present experimental results showing that this model improves the computational efficiency of Residual Networks on the challenging ImageNet classification and COCO object detection datasets. Additionally, we evaluate the computation time maps on the visual saliency dataset cat2000 and find that they correlate surprisingly well with human eye fixation positions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Neural Ordinary Differential Equations (2018, neutral): “Adaptive computation One can adapt computation time by training secondary neural networks to choose the number of evaluations of recurrent or residual networks (Graves, 2016; Jernite et al., 2016; Figurnov et al., 2017; Chang et al., 2018).”
Scalable Adaptive Computation for Iterative Generation (2022, background): “Other approaches for adaptive computation have mainly explored models with dynamic depth with recurrent networks (Graves, 2016; Figurnov et al., 2017) or sparse computation (Yin et al.”
Patient Knowledge Distillation for BERT Model Compression (2019, background): “focused on accelerating deep network inference at test time (Vetrov et al., 2017) and reducing model training time (Huang et al.”
Network Pruning via Transformable Architecture Search (2019, methodology): “Unstructured pruning methods [26, 7, 5, 9] usually enforce the convolutional weights [26, 11] or feature maps [5, 7] to be sparse.”
SkipNet: Learning Dynamic Routing in Convolutional Networks (2017, methodology): “We compare SkipNet with spatially adaptive computation time (SACT) and adaptive computation time (ACT) networks proposed by [5] on the ImageNet benchmark.”
FastBERT: a Self-distilling BERT with Adaptive Inference Time (2020, background): “…Conventional approaches in adaptive computations are performed token-wise or patch-wise, who either adds recurrent steps to individual tokens (Graves, 2016) or dynamically adjusts the number of executed layers inside discrete regions of images (Teerapittayanon et al., 2016; Figurnov et al., 2017).”
Adaptive Neural Networks for Efficient Inference (2017, background): “More recently, researchers have designed spatially adaptive networks (Figurnov et al., 2016a; Bengio et al., 2015) where nodes in a layer are selectively activated.”
Structured Bayesian Pruning via Log-Normal Multiplicative Noise (2017, background): “Spatial Skipped Convolutions [3] and Spatially Adaptive Computation Time [2] that propose different ways to reduce the number of computed convolutions, Binary Networks [18] that achieve speedup by using only 1 bit to store a single weight of a DNN, Low-Rank Expansions [6] that use low-rank filter ap”
Convolutional Networks with Adaptive Inference Graphs (2017, background): “Most closely related to our approach is the work on spatially adaptive computation time for residual networks (Figurnov et al. 2017).”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, methodology): “2 We compare to the following state-of-the-art methods 3: (1) ACT and (2) SACT [14], (3) PFEC [32], (4) LCCL [12].”
SpotTune: Transfer Learning Through Adaptive Fine-Tuning (2018, methodology): “Our proposed approach is related to conditional computation methods [4, 27, 11], which aim to dynamically route information in neural networks with the goal of improving computational efficiency.”
Efficient Segmentation: Learning Downsampling Near Semantic Boundaries (2019, background): “Other complementary approaches include skipping some layers at some pixels [17] and early stopping of network computation for some spatial regions of the image [16, 33].”
Learning Layer-Skippable Inference Network (2020, background): “Figurnov et al. [41] incorporated attention into ResNets for learning an image-dependent early-stop policy in residual units, both in layer level and feature map level.”
LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition (2019, positive): “Recently, the idea of conditional computation has also been investigated in deep neural networks [30, 15, 24, 6, 1, 22] through learning when to exit CNNs with attached decision branches.”
2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition (2020, methodology): “a, conditional computation) methods have been developed in the image domain, achieving reduced computation by dynamically selecting channels [2, 25], skipping layers [42, 11, 40, 37], performing early exiting with auxiliary structures [22, 18, 3, 46], adaptively switching input resolutions [27, 36, ”
SBNet: Sparse Blocks Network for Fast Inference (2018, background): “The network has no bottleneck layers and has one initial convolution layer, followed by [2, 2, 2, 2] residual units per residual blocks, with channel depth [32, 64, 128, 256].”
S2DNAS: Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search (2019, background): “Recently, dynamic inference has emerged as a promising alternative to speed up the CNN inference by dynamically changing the computation graph according to each input sample [40, 2, 6, 5, 46, 17, 29, 7].”
QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection (2021, neutral): “On the other hand, the Spatially Adaptive Computation time (SACT) [9] predicts a halting score for each spatial position that is supervised by a proposed ponder cost and the task-specific loss function.”