Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference

By Thomas Verelst, Tinne Tuytelaars. 2019

186 citations recorded by the source corpus.

Modern convolutional neural networks apply the same operations on every pixel in an image. However, not all image regions are equally important. To address this inefficiency, we propose a method to dynamically apply convolutions conditioned on the input image. We introduce a residual block where a small gating branch learns which spatial positions should be evaluated. These discrete gating decisions are trained end-to-end using the Gumbel-Softmax trick, in combination with a sparsity criterion. Our experiments on CIFAR, ImageNet, Food-101 and MPII show that our method has better focus on the region of interest and better accuracy than existing methods, at a lower computational complexity. Moreover, we provide an efficient CUDA implementation of our dynamic convolutions using a gather-scatter approach, achieving a significant improvement in inference speed on MobileNetV2 and ShuffleNetV2. On human pose estimation, a task that is inherently spatially sparse, the processing speed is increased by 60% with no loss in accuracy.

Subfields: Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, methodology): “elements on the output [144], [145]; and 3) estimating the saliency of pixels [146], [147], [148].”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, background): “Conditionally parameterized convolutions (Yang et al., 2019) or dynamic convolutions (Chen et al., 2019a; Verelst & Tuytelaars, 2019) have also been proposed to learn specialized convolutional kernels for each example to improve efﬁciency in image recognition.”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, background): “Other works focus on adaptive spatial attention for faster inference [14, 46, 47].”
BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policies (2021, background): “Recently, this has been extended to the spatial domain [7, 47, 40] by skipping individual pixels.”
LightGlue: Local Feature Matching at Light Speed (2023, background): “…p d A cross a ss i g n m e n t c c confidence nity [71, 80, 40, 21, 36, 76].”
Dynamic Region-Aware Convolution (2020, methodology): “[31, 34, 8] use dynamic inference to reduce computation time.”
Focal Sparse Convolutional Networks for 3D Object Detection (2022, background): “Methods of [39, 42, 45, 50] seek spatial-wise sparsity for efficient inference.”
QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection (2021, background): “Dynamic Convolution [52] uses a small gating network to predict pixel masks and [60] proposes a stochastic sampling and interpolation network.”