Dynamic Convolution: Attention Over Convolution Kernels

By Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, Zicheng Liu. 2019

1,399 citations recorded by the source corpus.

Light-weight convolutional neural networks (CNNs) suffer performance degradation as their low computational budgets constrain both the depth (number of convolution layers) and the width (number of channels) of CNNs, resulting in limited representation capability. To address this issue, we present Dynamic Convolution, a new design that increases model complexity without increasing the network depth or width. Instead of using a single convolution kernel per layer, dynamic convolution aggregates multiple parallel convolution kernels dynamically based upon their attentions, which are input dependent. Assembling multiple kernels is not only computationally efficient due to the small kernel size, but also has more representation power since these kernels are aggregated in a non-linear way via attention. By simply using dynamic convolution for the state-of-the-art architecture MobileNetV3-Small, the top-1 accuracy of ImageNet classification is boosted by 2.9% with only 4% additional FLOPs and 2.9 AP gain is achieved on COCO keypoint detection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, methodology): “To improve the representation power without noticeably increasing the computation, soft attention can be performed on multiple convolutional kernels, producing an adaptive ensemble of parameters [13], [19].”
Going deeper with Image Transformers (2021, methodology): “Various other attention mechanism have been used successfully to give a global view in conjunction with (local) convolutions [4, 52, 12, 77, 79], most mimic squeeze-and-excitate [32] for leveraging global features.”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “The mechanism is used channel-wise to capture cross-feature information that complements convolutional layers [39, 40, 41], select paths in different branch of a network [42], or combine both [43].”
Revisiting Dynamic Convolution via Matrix Decomposition (2021, methodology): “This is because a small attention score π (cid:100) i/C (cid:101) may suppress the learning of corresponding columns u i , v i in U and V , especially in early training epochs (as shown in Chen et al. (2020c)).”
Rewrite the Stars (2024, neutral): “Our analysis could offer new insights into the effectiveness of self-attention and contribute to the revisiting on ‘dynamic’ features [7, 8, 12] in neural networks.”
Large Selective Kernel Network for Remote Sensing Object Detection (2023, methodology): “CondConv [66] and Dynamic convolution [5] use parallel kernels to adaptively aggregate features from multiple convolution kernels.”
Focal Sparse Convolutional Networks for 3D Object Detection (2022, background): “, deformable convolutions [10, 64] and dynamic convolutions [7, 49].”
Attention mechanisms in computer vision: A survey (2021, background): “[116] proposed dynamic convolution, a novel operator design that increases representational power with negligible additional computational cost and does not change the width or depth of the network in parallel with CondConv [174].”
Focal Modulation Networks (2022, methodology): “Introducing dynamic operation is another way to augment ConvNets as demonstrated in Involution [37] and DyConv [10].”
HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions (2022, methodology): “(b) Dynamic convolution [27, 4] and SE [25] introduce the dynamic weights to improve the modeling power of convolutions with extra spatial interactions.”
Dynamic ReLU (2020, methodology): “5 as back-bone and use up-sampling and inverted residual bottleneck blocks [3] as head.”
Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition (2021, methodology): “We analyze the representation capability of different graph convolutions by reformulating them into a unified form and comparing them with dynamic convolution [2, 32] employed in CNNs.”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, background): “While GaterNet (Chen et al., 2019b) proposes a separate gating network to learn channel-wise binary gates for the backbone network, Channel gating network (Hua et al., 2019) identiﬁes regions in the features that contribute less to the classiﬁcation result, and skips the computation on a subset of…”
Dynamic DETR: End-to-End Object Detection with Dynamic Attention (2021, background): “[3] proposed to aggregate multiple parallel convolution kernels dynamically based upon their attentions, which had more representation power since these kernels are aggregated in a nonlinear way via attention.”
DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution (2020, background): “Conditional Convolution Conditional convolutional networks adopt dynamic kernels, widths, or depths, e.g ., [16,39,43,48,74,77].”
Omni-Dimensional Dynamic Convolution (2022, methodology): “Recent works (Wang et al., 2017; Hu et al., 2018b; Park et al., 2018; Woo et al., 2018; Yang et al., 2019; Chen et al., 2020) show that incorporating attention mechanisms into convolutional blocks can further push the performance boundaries of modern CNNs, and thus it has attracted great research…”
TAM: Temporal Adaptive Module for Video Recognition (2020, methodology): “Unlike some current dynamic convolution [3, 45], TAM is more flexible, and can directly generate the kernel weights to perform video adaptive convolution.”