AdderNet: Do We Really Need Multiplications in Deep Learning?

By Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian. 2019

236 citations recorded by the source corpus.

Compared with cheap addition operation, multiplication operation is of much higher computation complexity. The widely-used convolutions in deep neural networks are exactly cross-correlation to measure the similarity between input feature and convolution filters, which involves massive multiplications between float values. In this paper, we present adder networks (AdderNets) to trade these massive multiplications in deep neural networks, especially convolutional neural networks (CNNs), for much cheaper additions to reduce computation costs. In AdderNets, we take the L1-norm distance between filters and input feature as the output response. The influence of this new similarity measure on the optimization of neural network have been thoroughly analyzed. To achieve a better performance, we develop a special back-propagation approach for AdderNets by investigating the full-precision gradient. We then propose an adaptive learning rate strategy to enhance the training procedure of AdderNets according to the magnitude of each neuron's gradient. As a result, the proposed AdderNets can achieve 74.9% Top-1 accuracy 91.7% Top-5 accuracy using ResNet-50 on the ImageNet dataset without any multiplication in convolutional layer. The codes are publicly available at: (https://github.com/huaweinoah/AdderNet).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Margin-Based Few-Shot Class-Incremental Learning with Class-Level Overfitting Mitigation (2022, background): “Moreover, each convolution kernel, as a pattern extractor, can be understood as a semantic template [5], and the activation on each channel can be viewed as the matching score between the template and the input.”
COVID-19 Detection Through Transfer Learning Using Multimodal Imaging Data (2020, background): “Recent studies attempted to improve performance of the base models by proposing new layers and ﬁlters such as Sparse Shift Filter [77], Asymmetric Convolution Block [78], Adder Networks [79], Virtual Pooling [80], Discrete Wavelet Transform [81], and HetConv [82], etc.”
End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things (2020, methodology): “We can also consider training convolutional neural networks using addition operations only without multiplication operations based on the previous study results [30] since multiplication operations are significantly computationally heavier than addition operations in CNN models.”
Rewrite the Stars (2024, unknown): “Theoretically, multiplication operations (such as the star operation in our study) are understood to have a higher computational complexity compared to simpler summation operations, as indicated in several related works [5, 15].”
Progressive Network Grafting for Few-Shot Knowledge Distillation (2020, unknown): “Resuing pre-trained networks has recently attracted attentions from researchers in the field (Chen et al. 2020; Yu et al. 2017).”
Revisiting Dynamic Convolution via Matrix Decomposition (2021, background): “Chen et al. (2020a) trades massive multiplications for much cheaper additions.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “AdderNet [35] is another technique that replaces multiply arithmetic but allows larger than 1-bit parameters.”