Continuously Differentiable Exponential Linear Units

By Jonathan T. Barron. 2017

169 citations recorded by the source corpus.

Exponential Linear Units (ELUs) are a useful rectifier for constructing deep learning architectures, as they may speed up and otherwise improve learning by virtue of not have vanishing gradients and by having mean activations near zero. However, the ELU activation as parametrized in [1] is not continuously differentiable with respect to its input when the shape parameter alpha is not equal to 1. We present an alternative parametrization which is C1 continuous for all values of alpha, making the rectifier easier to reason about and making alpha easier to tune. This alternative parametrization has several other useful properties that the original parametrization of ELU does not: 1) its derivative with respect to x is bounded, 2) it contains both the linear transfer function and ReLU as special cases, and 3) it is scale-similar with respect to alpha.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Mei
2. Alan L. Yuille
3. Quoc V. Le
4. Ting Yao
5. Yingwei Pan
6. Fahad Shahbaz Khan
7. Yehao Li
8. Boqing Gong

In Control and Systems Engineering

1. Mingxing Tan
2. Fei Deng
3. Weihao Sun
4. Tao Mei
5. Ibrahim Radwan
6. Alan L. Yuille
7. Ting Yao
8. Mohammed Hassanin

Citations in the corpus

• SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition (2020, methodology): “Input 3 Conv 3× 3 16 1 GN(4)/CELU Conv 4× 4 32 2 GN(8)/CELU Conv 3× 3 32 1 GN(4)/CELU Conv 4× 4 64 1 GN(8)/CELU Conv 4× 4 128 2 GN(8)/CELU Conv 4× 4 256 1 GN(16)/CELU Linear 32 + 32 Glimpse Decoder Layer Size/Ch. Stride Norm.”
• Generative Neurosymbolic Machines (2020, methodology): “Note that in Table 9, all convolutional and MLP layers except the output layers are followed by a CELU activation function [4] and a layer normalization [2].”
• Smooth Adversarial Training (2020, methodology): “To stabilize the adversarial training with ELU, we apply its smooth version, CELU [3], which re-parametrize ELU to the following format: Note that CELU is equivalent to ELU when α = 1 .”
• Activation functions in deep learning: A comprehensive survey and benchmark (2021, background): “The ReLU, LReLU, ELU, GELU, CELU, ABReLU, and PDELU activation functions are better for the networks having residual connections, such as ResNet50, SENet18 and DenseNet121.”
• Visual Attention Methods in Deep Learning: An In-Depth Survey (2022, methodology): “Modeling infinity-order interaction is also explained using h Exponential Linear Unit [9].”
• X-Linear Attention Networks for Image Captioning (2020, methodology): “Higher and even infinity order feature interactions are readily modeled through stacking multiple X-Linear attention blocks and equipping the block with Exponential Linear Unit (ELU) in a parameter-free fashion, respectively.”
• RNA secondary structure prediction using deep learning with thermodynamic integration (2020, methodology): “The layer normalization (20) and the continuously differentiable exponential linear units (CELU) activation function (21) are employed.”
• NPENAS: Neural Predictor Guided Evolution for Neural Architecture Search (2022, methodology): “The graph-based uncertainty estimation network employs CELU [50] activation function, and the graph-based neural predictor employs ReLU activation function.”