Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation

By Zhenda Xie, Zheng Zhang, Xizhou Zhu, Gao Huang, Stephen Lin. 2020

103 citations recorded by the source corpus.

In the feature maps of CNNs, there commonly exists considerable spatial redundancy that leads to much repetitive processing. Towards reducing this superfluous computation, we propose to compute features only at sparsely sampled locations, which are probabilistically chosen according to activation responses, and then densely reconstruct the feature map with an efficient interpolation procedure. With this sampling-interpolation scheme, our network avoids expending computation on spatial locations that can be effectively interpolated, while being robust to activation prediction errors through broadly distributed sampling. A technical challenge of this sampling-based approach is that the binary decision variables for representing discrete sampling locations are non-differentiable, making them incompatible with backpropagation. To circumvent this issue, we make use of a reparameterization trick based on the Gumbel-Softmax distribution, with which backpropagation can iterate these variables towards binary values. The presented network is experimentally shown to save substantial computation while maintaining accuracy over a variety of computer vision tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Gao Huang
2. Zheng Zhang
3. Yue Cao
4. Shiji Song
5. Han Hu
6. Dacheng Tao
7. Jiaya Jia
8. Jian‐Yuan Sun

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Ming-Hsuan Yang
4. Alan L. Yuille
5. Xiaogang Wang
6. Trevor Darrell
7. Jitendra Malik
8. Yi Yang

In Efficiency & Compression

1. Shiji Song
2. Gao Huang
3. Yulin Wang
4. Le Yang
5. Yizeng Han
6. Honghui Wang
7. Naiyan Wang
8. Zehao Huang

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, methodology): “, spatially adaptive models [33], [39], [148] and soft”
• Adaptive Focus for Efficient Video Recognition (2021, background): “For example, in 2D-image classification, a number of recent works successfully improve the efficiency of CNNs by attending to some task-relevant or more informative parts of images [47, 41, 13].”
• Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification (2020, methodology): “Recent research has revealed that considerable spatial redundancy occurs when inferring CNNs [10, 62].”
• Focal Sparse Convolutional Networks for 3D Object Detection (2022, methodology): “Methods of [39, 42, 45, 50] seek spatial-wise sparsity for efficient inference.”
• QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection (2021, background): “Dynamic Convolution [52] uses a small gating network to predict pixel masks and [60] proposes a stochastic sampling and interpolation network.”
• SimMIM: a Simple Framework for Masked Image Modeling (2021, background): “Such claim is also partly supported by recent works of sparse inference [19] that the recognition accuracy has very little drop after throwing a large portion of image features [23,41,51].”