Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks

By Tomasz Szandala. Studies in Computational Intelligence, 2020

390 citations recorded by the source corpus.

The primary neural networks’ decision-making units are activation functions. Moreover, they evaluate the output of networks neural node; thus, they are essential for the performance of the whole network. Hence, it is critical to choose the most appropriate activation function in neural networks calculation. Acharya et al. (2018) suggest that numerous recipes have been formulated over the years, though some of them are considered deprecated these days since they are unable to operate properly under some conditions. These functions have a variety of characteristics, which are deemed essential to successfully learning. Their monotonicity, individual derivatives, and finite of their range are some of these characteristics. This research paper will evaluate the commonly used additive functions, such as swish, ReLU, Sigmoid, and so forth. This will be followed by their properties, own cons and pros, and particular formula application recommendations.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/be753166d1613e3c030a5b03823146b03ea40953

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bidyut Baran Chaudhuri
2. Satish Kumar Singh
3. Shiv Ram Dubey

Citations in the corpus

• Activation functions in deep learning: A comprehensive survey and benchmark (2021, methodology): “The evaluation of seven AFs is conducted in [147] using a simple network over CIFAR10 dataset, whereas in our survey we cover different AFs and also perform the experimental comparison.”