Activation Functions: Comparison of trends in Practice and Research for Deep Learning

By Chigozie Nwankpa, Winifred L. Ijomah, Anthony Gachagan, Stephen Marshall. arXiv.org, 2018

1,459 citations recorded by the source corpus.

Deep neural networks have been successfully used in diverse emerging domains to solve real world complex problems with may more deep learning(DL) architectures, being developed to date. To achieve these state-of-the-art performances, the DL architectures use activation functions (AFs), to perform diverse computations between the hidden layers and the output layers of any given DL architecture. This paper presents a survey on the existing AFs used in deep learning applications and highlights the recent trends in the use of the activation functions for deep learning applications. The novelty of this paper is that it compiles majority of the AFs used in DL and outlines the current trends in the applications and usage of these functions in practical deep learning deployments against the state-of-the-art research results. This compilation will aid in making effective decisions in the choice of the most suitable and appropriate activation function for any given application, ready for deployment. This paper is timely because most research papers on AF highlights similar works and results while this paper will be the first, to compile the trends in AF applications in practice against the research results from literature, found in deep learning research to date.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/594a91fe54eb44f0e5fffd35ab015a8456419d11

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks (2020, positive): “It is noted that for simple regression problems with neural networks, tanh can be superior to ReLU[34].”
Activation functions in deep learning: A comprehensive survey and benchmark (2021, background): “In contrast to [138], this paper presents a comprehensive survey of AFs.”
A review of convolutional neural networks in computer vision (2024, positive): “ReLU and its variants, on the other hand, are preferred because they aid in overcoming the vanishing gradient problem (Nwankpa et al. 2018).”
Model complexity of deep learning: a survey (2021, background): “Similarly, for a hard Tanh neural network under the same initialization, the trajectory length is bounded by E[l(z(i)(x(t)))] ≥ O ⎛ ⎜⎜⎝ σw √ m √ σ 2w + σ 2b + m √ σ 2w + σ 2b ⎞ ⎟⎟⎠ i l(x(t)).”
How Artificial Intelligence Is Shaping Medical Imaging Technology: A Survey of Innovations and Applications (2023, methodology): “For classification tasks, a sigmoid or a softmax activation function is typically used to calculate class probabilities, providing the final output of the CNN [25,26].”
A review on weight initialization strategies for neural networks (2021, background): “Nwankpa et al. (2018) have presented a comparison of various activation functions like sigmoid and its variants, rectified linear unit (ReLU) and its variants, hyperbolic tangent (tanh), softmax, softsign, softplus, exponential linear unit (ELU) and its variants, maxout, swish, exponential linear…”
A Review on Deep Learning in UAV Remote Sensing (2021, background): “2) are generally composed of dense layers, wherein activation functions are implemented in. Activation functions compute the weighted sum of input and biases, which is used to decide if a neuron can be activated or not [141].”
Solving Allen-Cahn and Cahn-Hilliard Equations using the Adaptive Physics Informed Neural Networks (2020, methodology): “In deep learning continuous activation functions are more commonly used [19].”
A survey on modern trainable activation functions (2020, background): “Apart from works which make an explicit comparison with other trainable (and not) activation functions (DasGupta and Schnitger, 1993; Karlik and Olgac, 2011; Nwankpa et al., 2018), it is difficult to make a complete comparison between all the existing architectures, even if the experiments are usual”
Recent advances and applications of deep learning methods in materials science (2021, background): “57 .”
Cancer Diagnosis Using Deep Learning: A Bibliographic Review (2019, methodology): “(iii) Rectiﬁed linear unit (ReLU) is the most commonly used activation function [45–47], where g denotes pixel-wise function, which is nonlinear in nature.”
A survey of the recent architectures of deep convolutional neural networks (2019, background): “However, ReLU and its variants are preferred as they help in overcoming the vanishing gradient problem (Hochreiter 1998; Nwankpa et al. 2018).”
SMYRF: Efficient Attention using Asymmetric Clustering (2020, background): “Since our goal is to demonstrate the ability of SMYRF layers to train successfully from scratch, there is no reason to use higher (image) resolutions than the attention resolution and thus SMYRF is the final layer (before Tanh [77]) in the architecture.”
Self-adaptive loss balanced Physics-informed neural networks (2021, background): “where σ is the activation function including sigmoid, relu, and tanh [51,52].”