Revisiting Dynamic Convolution via Matrix Decomposition

By Yunsheng Li, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Ye Yu, Lu Yuan, Zicheng Liu, Mei Chen, Nuno Vasconcelos. 2021

86 citations recorded by the source corpus.

Recent research in dynamic convolution shows substantial performance boost for efficient CNNs, due to the adaptive aggregation of K static convolution kernels.It has two limitations: (a) it increases the number of convolutional weights by K-times, and (b) the joint optimization of dynamic attention and static convolution kernels is challenging. In this paper, we revisit it from a new perspective of matrix decomposition and reveal the key issue is that dynamic convolution applies dynamic attentions over channel groups after projecting into a higher dimensional intermediate space. To address this issue, we propose dynamic channel fusion to replace dynamic attentions over channel groups. Dynamic channel fusion not only enables significant dimension reduction of the intermediate space, but also mitigates the joint optimization difficulty. As a result, our method is easier to train and requires significantly fewer parameters without sacrificing accuracy.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Chunwei Tian
2. Kai Zhu
3. Aojun Zhou
4. Yang Cao
5. Wei Zhai
6. Zhengjun Zha
7. Wangmeng Zuo
8. Jiebo Luo

Citations in the corpus

• Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning (2022, background): “Further improvements have been investigated for superior dynamic performance [19] and inference efficiency [5, 6, 9] recently.”
• Dynamic Slimmable Network (2021, unknown): “In some of dynamic networks, such as dynamic pruning [29, 14] and conditional convolution [68, 45], the convolution filters W are conditionally parameterized by a function A ( θ, X ) to the input X .”
• Omni-Dimensional Dynamic Convolution (2022, methodology): “In the experiments, we consider existing dynamic convolution methods including CondConv (Yang et al., 2019), DyConv (Chen et al., 2020) and DCD (Li et al., 2021) as the key reference methods for comparisons on all CNN backbones.”
• Multi-stage image denoising with the wavelet transform (2022, methodology): “Besides, using matrix decomposition to guide a dynamic convolution can make a tradeoff between performance and training speed in image recognition [17].”