Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
By Djork-Arné Clevert, Thomas Unterthiner, Sepp Hochreiter. 2015
6,140 citations recorded by the source corpus.
We introduce the "exponential linear unit" (ELU) which speeds up learning in deep neural networks and leads to higher classification accuracies. Like rectified linear units (ReLUs), leaky ReLUs (LReLUs) and parametrized ReLUs (PReLUs), ELUs alleviate the vanishing gradient problem via the identity for positive values. However, ELUs have improved learning characteristics compared to the units with other activation functions. In contrast to ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero like batch normalization but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit natural gradient because of a reduced bias shift effect. While LReLUs and PReLUs have negative values, too, they do not ensure a noise-robust deactivation state. ELUs saturate to a negative value with smaller inputs and thereby decrease the forward propagated variation and information. Therefore, ELUs code the degree of presence of particular phenomena in the input, while they do not quantitatively model the degree of their absence. In experiments, ELUs lead not only to faster learning, but also to significantly better generalization performance than ReLUs and LReLUs on networks with more than 5 layers. On CIFAR-100 ELUs networks significantly outperform ReLU networks with batch normalization while batch normalization does not improve ELU networks. ELU networks are among the top 10 reported CIFAR-10 results and yield the best published result on CIFAR-100, without resorting to multi-view evaluation or model averaging. On ImageNet, ELU networks considerably speed up learning compared to a ReLU network with the same architecture, obtaining less than 10% classification error for a single crop, single model network.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Quoc V. Le
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Ruslan Salakhutdinov
- Kyunghyun Cho
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In Model Architectures
- Quoc V. Le
- Yoshua Bengio
- Ruslan Salakhutdinov
- Alan L. Yuille
- Barret Zoph
- Kyunghyun Cho
- Aaron C. Courville
- Mohit Bansal
In Optimization & Training
- Yoshua Bengio
- Quoc V. Le
- Frank Hutter
- Stefano Soatto
- Simon Lacoste-Julien
- Ruslan Salakhutdinov
- Sergey Levine
- Trevor Darrell
Citations in the corpus
- Xception: Deep Learning with Depthwise Separable Convolutions (2016, methodology): “However we also experimentally tested the inclusion of either ReLU or ELU [3] as intermediate non-linearity.”
- Gaussian Error Linear Units (GELUs) (2016, background): “Meanwhile ReLUs and ELUs, which are convex and monotonic activations, are linear in the positive domain and thereby can lack curvature.”
- Wide Residual Networks (2016, neutral): “rameters than ResNet-1001 and both outperform it by a significant margin. depth-k # params CIFAR-10 CIFAR-100 NIN [20] 8.81 35.67 DSN [19] 8.22 34.57 FitNet [24] 8.39 35.04 Highway [28] 7.72 32.39 ELU [5] 6.55 24.28 original-ResNet[11] 110 1.7M 6.43 25.16 1202 10.2M 7.93 27.82 stoc-depth[14] 110 1.7M”
- Rethinking Attention with Performers (2020, methodology): “We use the attention implementation of the Linear Transformer from (Katharopoulos et al., 2020), which mainly involves setting our feature map φ ( x ) = elu ( x ) + 1 , where elu ( x ) is the shifted-eLU function from (Clevert et al., 2016).”
- HyperNetworks (2016, background): “…2014) 8.81% FitNet (Romero et al., 2014) 8.39% Deeply Supervised Nets (Lee et al., 2015) 8.22% Highway Networks (Srivastava et al., 2015) 7.72% ELU (Clevert et al., 2015) 6.55% Original Resnet-110 (He et al., 2016a) 6.43% 1.7 M Stochastic Depth Resnet-110 (Huang et al., 2016b) 5 parameters in the…”
- Improved Techniques for Training Score-Based Generative Models (2020, methodology): “We use the ELU activation function [18] throughout all architectures.”
- Improved Variational Inference with Inverse Autoregressive Flow (2016, methodology): “We use either diagonal Gaussian posteriors, or IAF with a single step of convolutional MADE van den Oord et al. (2016) with zero, one or two layers of hidden layers with ELU nonlinearities Clevert et al. (2015).”
- Identity Mappings in Deep Residual Networks (2016, methodology): “Comparisons with state-of-the-art methods on CIFAR-10 and CIFAR-100 using “moderate data augmentation” (flip/translation), except for ELU [12] with no augmentation.”
- Thinking Fast and Slow with Deep Learning and Tree Search (2017, methodology): “Layers 9 and 10 do not pad their input, and layers 11 and 13 do not pad, and have 1 × 1 filters Our convolution layers use Exponential linear unit (ELU) [19] nonlinearities.”
- Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields (2017, methodology): “To make results comparable with [31], the Coulomb GAN used a discriminator network with 2 hidden layers of 128 units, however we avoided batch normalization by using the ELU activation function [8].”
- f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization (2016, methodology): “The variational function is the same as the discriminator architecture in [27] and follows the structure of a convolutional neural network with batch normalization, exponential linear units [4] and one final linear layer.”
- Planning chemical syntheses with deep neural networks and symbolic AI (2017, methodology): “[35, 36] To assess its generalisation ability, we performed a time-split strategy.”
- Value Prediction Network (2017, methodology): “Exponential linear unit (ELU) [5] was used as an activation function for all architectures.”
- Learning Transferable Architectures for Scalable Image Recognition (2017, methodology): “We also tried ELU [10] instead of ReLUs and found that performance was about the same.”
- Do Transformer Modifications Transfer Across Implementations and Applications? (2021, methodology): “, 2017), (3) Exponential Linear Units (ELU) (Clevert et al., 2015), (4) Scaled exponential linear units (SeLU) (Klambauer et al.”
- Demystifying MMD GANs (2018, background): “Other activation functions, such as the ELU (Clevert et al., 2016), are piecewise-analytic and also satisfy Assumptions C and D .”