Dynamic ReLU

By Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, Zicheng Liu. 2020

221 citations recorded by the source corpus.

Rectified linear units (ReLU) are commonly used in deep neural networks. So far ReLU and its generalizations (non-parametric or parametric) are static, performing identically for all input samples. In this paper, we propose dynamic ReLU (DY-ReLU), a dynamic rectifier of which parameters are generated by a hyper function over all in-put elements. The key insight is that DY-ReLU encodes the global context into the hyper function, and adapts the piecewise linear activation function accordingly. Compared to its static counterpart, DY-ReLU has negligible extra computational cost, but significantly more representation capability, especially for light-weight neural networks. By simply using DY-ReLU for MobileNetV2, the top-1 accuracy on ImageNet classification is boosted from 72.0% to 76.2% with only 5% additional FLOPs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, neutral): “For instance, DY-ReLU [132] replaces ReLU (yc = max(xc, 0)) with the max value among N linear transformations yc = maxn {ac xc + bc }, where c is the channel index, and ac , bc are linear coefficients calculated from x.”
Rewrite the Stars (2024, unknown): “Our analysis could offer new insights into the effectiveness of self-attention and contribute to the revisiting on ‘dynamic’ features [7, 8, 12] in neural networks.”
MobileOne: An Improved One millisecond Mobile Backbone (2022, methodology): “This can be attributed to synchronization costs mostly incurred by recently introduced activation functions like SE-ReLU [32], Dynamic Shift-Max [36] and Dynami-cReLUs [6].”
Activation functions in deep learning: A comprehensive survey and benchmark (2021, methodology): “In a very similar approach, the maximum of multiple linear functions is used in the Dynamic ReLU (DYReLU) [114].”
Lite-HRNet: A Lightweight High-Resolution Network (2021, background): “Existing efficient networks [5, 6, 50] are mainly designed from two perspectives.”
MetaFormer Baselines for Vision (2022, methodology): “To make the activation adaptable to different situations, like different models or initialization, scale and bias can be set to be learnable [23, 8].”