Learning Activation Functions to Improve Deep Neural Networks

By Forest Agostinelli, Matthew D. Hoffman, Peter Sadowski, Pierre Baldi. 2014

512 citations recorded by the source corpus.

Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural network architectures composed of static rectified linear units, achieving state-of-the-art performance on CIFAR-10 (7.51%), CIFAR-100 (30.83%), and a benchmark from high-energy physics involving Higgs boson decay modes.

Subfields: Artificial Intelligence, Geophysics, Radiology, Nuclear Medicine and Imaging, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Geophysics

In Radiology, Nuclear Medicine and Imaging

In Model Architectures

Citations in the corpus

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, neutral): “2Concurrently, [1] also investigated learning activations.”
Deep Networks with Stochastic Depth (2016, methodology): “Horizontal ﬂipping and translation by 4 pixels are the two standard data augmentation techniques adopted in our experiments, following the common practice [6, 13, 20, 21, 23, 25, 29].”
Swish: a Self-Gated Activation Function (2017, background): “While much work has focused on proposing new activation functions (Maas et al., 2013; Agostinelli et al., 2014; He et al., 2015; Clevert et al., 2015; Klambauer et al., 2017; Qiu & Cai, 2017), few studies, such as Xu et al. (2015), have systematically compared different activation functions.”
Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree (2015, background): “We observe that the increase in the number of parameters is not met with a corresponding boost in performance, and so we pursue the “one per layer” option.”
Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks (2016, methodology): “…DSN (Lee et al., 2015) 34.57 Maxout (Goodfellow et al., 2013) 38.57 with data augmentation NormProp 29.24 Batch Normalization 30.26 NIN + ALP units (Agostinelli et al., 2015) 30.83 NIN (Lin et al., 2014) - DSN (Lee et al., 2015) - Maxout (Goodfellow et al., 2013) - 1.88 Batch Normalization 2.25…”
Activation functions in deep learning: A comprehensive survey and benchmark (2021, background): “The Adaptive Piecewise Linear (APL) is defined as a sum of hinge-shape functions [28].”
Adversarial Attacks on Graph Neural Networks via Meta Learning (2019, unknown): “…or learning to learn , is the task of optimizing the learning algorithm itself; e.g., by optimizing the hyperparameters Bengio (2000), learning to update the parameters of a neural network (Schmidhuber, 1992; Bengio et al., 1992), or the activation function of a model (Agostinelli et al., 2014).”
Learning Optimal Nonlinearities for Iterative Thresholding Algorithms (2015, background): “A general application of deep learning to compressive sensing was presented in [26], while the idea of learning the activation functions was discussed in [27].”
Inducing and Exploiting Activation Sparsity for Fast Neural Network Inference (2020, background): “Agostinelli et al. (2014) propose learning piecewise linear activation functions to improve the accuracy of given models.”
Efficient Continual Learning in Neural Networks with Embedding Regularization (2019, background): “Several papers recently have shown that, in the single-task scenario, training the activation functions [8, 1, 11, 22, 26] can provide significant improvements in performance and in the number of layers required to solve the given task.”
Generalizing Pooling Functions in CNNs: Mixed, Gated, and Tree (2018, unknown): “92 NiN + LA units [41] 9.”
Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration (2015, background): “There are actually two recent works [2], [29] to investigate a parameterized ReLU function in standard deep convolutional networks, which indeed brings improvements even with little freedom to tune the activation functions.”
Deep Learning for Change Detection in Remote Sensing Images: Comprehensive Review and Meta-Analysis (2020, background): “Typically, after the convolution operation a nonlinear activation function is performed on each element of the convolution result. ykl = f ( zkl ) (11) A range of activation functions has been proposed in the literature to improve the performance of CNNs, for example the sigmoid function [49], hyper”
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review (2017, background): “-100 SV-HN DropConnect (Wan et al., 2013) Optimized DCNNs with dropped connections 18.7 9.32 - 1.94 Stochastic pooling (Zeiler & Fergus, 2013) Stochastic rather than deterministic pooling procedure 15.13 - 42.51 2.80 Maxout networks (Goodfellow et al., 2013) Maxout activation functions 11.68 9.38 38”
Searching for Activation Functions (2018, background): “Most prior work has focused on proposing new activation functions (Maas et al., 2013; Agostinelli et al., 2014; He et al., 2015; Clevert et al., 2015; Hendrycks & Gimpel, 2016; Klambauer et al., 2017; Qiu & Cai, 2017; Zhou et al., 2017; Elfwing et al., 2017), but few studies, such as Xu et al.”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, background): “Some of the common nonlinear functions are rectiﬁed nonlinear unit (ReLU), leaky ReLU, sigmoid, Tanh, and ELU [44].”
Optimization Models for Machine Learning: A Survey (2019, background): “In [1], adaptive piecewise linear activation functions are learned when training each neuron.”