Performance Analysis of Various Activation Functions in Generalized MLP Architectures of Neural Networks

By B. Karlik, A. V. Olgaç. 2011

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations (2020, methodology): “To tackle this challenge, we conduct BatchNorm [11] with the Tanh activation [13] on the transformed embeddings {̃e1 i , · · · , ẽ N i } as follows:”
Privacy-preserving Machine Learning as a Service (2018, neutral): “This approach was previously used in [26] for comparing performances of different activation functions.”
A novel hybrid extreme learning machine–grey wolf optimizer (ELM-GWO) model to predict compressive strength of concrete with partial replacements for cement (2020, methodology): “The tangent hyperbolic function is an activation function that generally leads to more accurate results [36]; thus, this activation function is used in this study.”
Genetic deep neural networks using different activation functions for financial data mining (2015, background): “For instance, among the activation functions (bipolar sigmoid, unipolar sigmoid, hyperbolic tangent, conic section, and radial bases functions), simulation results indicate that the hyperbolic tangent function performs the best [4].”
MR fingerprinting Deep RecOnstruction NEtwork (DRONE) (2017, methodology): “The sigmoid and tanh activation functions used in this study are a common choice for NN training 35 but skew the accuracy of the network toward the middle of the training dictionary range (Figure 2), where the gradient is largest and the back-propagation algorithm is thus most effective.”
A survey on modern trainable activation functions (2020, background): “Apart from works which make an explicit comparison with other trainable (and not) activation functions (DasGupta and Schnitger, 1993; Karlik and Olgac, 2011; Nwankpa et al., 2018), it is difficult to make a complete comparison between all the existing architectures, even if the experiments are usual”
Prediction of self-compacting concrete strength using artificial neural networks (2016, background): “Past studies (Bartlett, 1998; Karlik & Olgac, 2011) have proposed a large number of alternative transfer functions.”
Activation functions in deep learning: A comprehensive survey and benchmark (2021, unknown): “Karlik and Olgac [134] Multilayer Perceptron (MLP) 5 AFs, including Bi-polar sigmoid, Uni-polar sigmoid, Tanh, etc.”
Concrete compressive strength using artificial neural networks (2019, background): “A large variety of alternative transfer functions have been proposed in the relevant literature [45, 46].”
Activation Functions: Comparison of trends in Practice and Research for Deep Learning (2018, background): “These AFs are often referred to as a transfer function in the literature, with early research results by [15], validating categorically that a proper choice of AF improves results in NN computing.”
Deep Learning on Image Denoising: An overview (2019, unknown): “The activation functions such as Sigmoid [154, 104] and tanh [92, 55] can convert the linear input into non-linearity through W and b as follows. f Note that if the neural network has multiple layers, it is regarded as multilayer perceptron (MLP) [25].”