Empirical Evaluation of Rectified Activations in Convolutional Network

By Bing Xu, Naiyan Wang, Tianqi Chen, Mu Li. 2015

3,198 citations recorded by the source corpus.

In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evaluate these activation function on standard image classification task. Our experiments suggest that incorporating a non-zero slope for negative part in rectified activation units could consistently improve the results. Thus our findings are negative on the common belief that sparsity is the key of good performance in ReLU. Moreover, on small scale dataset, using deterministic negative slope or learning it are both prone to overfitting. They are not as effective as using their randomized counterpart. By using RReLU, we achieved 75.68\% accuracy on CIFAR-100 test set without multiple test or ensemble.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, background): “Another variant are Randomized Leaky Rectified Linear Units (RReLUs) which randomly sample the slope of the negative part which raised the performance on image benchmark datasets and convolutional networks (Xu et al., 2015).”
Scalable agent alignment via reward modeling: a research direction (2018, background): “…modern deep networks typically use piece-wise linear activation functions (Nair & Hinton, 2010; Glorot et al., 2011; Goodfellow et al., 2013; Xu et al., 2015), which generalize linearly far from training data (Goodfellow et al., 2015), meaning estimated reward would go to positive or…”
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (2015, methodology): “Similarly, using a gradient descent on the inputs lets us inspect the ideal image that activates certain subsets of ﬁlters (Mordvintsev et al.).”
Deep Reinforcement Learning that Matters (2017, background): “Furthermore, certain activation functions such as Rectiﬁed Linear Unit (ReLU) have been shown to cause worsened learning performance due to the “dying relu” problem (Xu et al. 2015).”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, background): “The Stage-II GAN completes previously ignored text information to generate more photo-realistic details.”
Swish: a Self-Gated Activation Function (2017, background): “While much work has focused on proposing new activation functions (Maas et al., 2013; Agostinelli et al., 2014; He et al., 2015; Clevert et al., 2015; Klambauer et al., 2017; Qiu & Cai, 2017), few studies, such as Xu et al. (2015), have systematically compared different activation functions.”
Generative Image Modeling Using Style and Structure Adversarial Networks (2016, methodology): “Negative slope for LeakyReLU is 0.2.”
Learning to Discover Cross-Domain Relations with Generative Adversarial Networks (2017, methodology): “The encoder part of each generator is composed of convolution layers with 4 × 4 ﬁl-ters, each followed by leaky ReLU (Maas et al., 2013; Xu et al., 2015).”
Recurrent Environment Simulators (2017, methodology): “Every layer was followed by a randomized rectified linear function (RReLU) (Xu et al., 2015) with parameters l = 1/8, u = 1/3.”
Compositional Fairness Constraints for Graph Embeddings (2019, methodology): “In all experiments, we used multi-layer perceptrons (MLPs) with leaky ReLU activation functions (Xu et al., 2015) as the discriminators D k and ﬁlters f k .”
Few-Shot Learning with Graph Neural Networks (2017, positive): “A GNN layer Gc ( · ) receives as input a signal where Θ = { θ B ∈ R d k × d k +1 , are trainable parameters and ρ ( · ) is a point-wise non-linearity, chosen in this work to be a ‘leaky’ ReLU Xu et al. (2015).”
High-Fidelity Generative Image Compression (2020, background): “Norm is ChannelNorm (see text), LReLU the leaky ReLU [56] with α=0.”
In-place Activated BatchNorm for Memory-Optimized Training of DNNs (2017, methodology): “However, we can replace it with LEAKY RELU and a small slope without impacting the quality of the trained models [31].”
Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units (2016, methodology): “Another way to allow negative activation is to employ the broader class of non-saturated activation functions including Leaky Relu and its variants (Xu et al., 2015; Maas et al., 2013).”
Neural Autoregressive Flows (2018, background): “One possibility is using other (strictly) monotonic activation functions in τ c , such as leaky ReLUs (LReLU) (Xu et al., 2015) or ELUs (Clevert et al., 2015).”
Speech-to-Speech Translation Between Untranscribed Unknown Languages (2019, methodology): “Residual 1D block combines multiscale 1D convolution with different kernel size and “SAME” padding, LeakyReLU [25] activation function, and batch normalization [26].”
FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters (2015, background): “[32] have each shown that seemingly-subtle hyperparameter settings such as weight initialization can have a big impact on the speed and accuracy produced in DNN training.”