Deep Sparse Rectifier Neural Networks

By Xavier Glorot, Antoine Bordes, Yoshua Bengio. 2011

8,901 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Generative Adversarial Nets (2014, methodology): “These striking successes have primarily been based on the backpropagation and dropout algorithms, using piecewise linear units [17, 8, 9] which have a particularly well-behaved gradient .”
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, methodology): “The concatenated vector was fed through two rectified layers (of size 1536 and 1024) (Glorot et al., 2011).”
A Decomposable Attention Model for Natural Language Inference (2016, methodology): “We take F to be a feed-forward neural network with ReLU activations (Glorot et al., 2011).”
On the Number of Linear Regions of Deep Neural Networks (2014, methodology): “1.5 On the other hand, recently it has become more common to use piecewise linear functions, such as the rectiﬁer activation g ( a ) = max 0 , a (Glorot et al. 2011, Nair and Hinton 2010) or the maxout activation g ( a 1 , . . . 1.0 These previous theoretical results, however, do not trivially…”
Maxout Networks (2013, background): “Note that the arguments in section 7 motivating the use of maxout also apply equally to rectiﬁed linear units (Salinas & Abbott, 1996; Hahnloser, 1998; Glorot et al., 2011).”
Pylearn2: a machine learning research library (2013, methodology): “Pylearn2 has implementations of several models that were de v loped at LISA, including denoising auto-encoders (DAEs) [53], contractive auto-encoders (CAEs) [42] including higher-order CAEs [43], spike-and-slab RBMs (ssRBMs) [11] including ssR BMs with pooling [12], reconstruction sampling autoencod”
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation (2013, methodology): “…condition for exact gradient-based training methods to work well, recent successes of deep networks with rectiﬁers and other “non-smooth” non-linearities (Glorot et al. , 2011; Krizhevsky et al. , 2012a; Goodfellow et al. , 2013) clearly question that belief: see Section 2 for a deeper discussion.”
Adversarial Training Methods for Semi-Supervised Text Classification (2016, background): “The activation function on the hidden layer was ReLU(Jarrett et al., 2009; Nair & Hinton, 2010; Glorot et al., 2011).”
GLU Variants Improve Transformer (2020, background): “A rectified-linear (ReLU) [Glorot et al., 2011] activation function applied between the two linear transformations.”
Explaining and Harnessing Adversarial Examples (2014, background): “LSTMs (Hochreiter & Schmidhuber, 1997), ReLUs (Jarrett et al., 2009; Glorot et al., 2011), and maxout networks (Goodfellow et al.”
Spectral Normalization for Generative Adversarial Networks (2018, positive): “The slopes of all lReLU functions in the networks are set to 0 .”
Learning to Generate Samples from Noise through Infusion Training (2017, methodology): “We use rectifier linear units (Glorot et al., 2011) on each layer inside the networks.”
Conditional Generative Adversarial Nets (2014, methodology): “Both z and y are mapped to hidden layers with Rectified Linear Unit (ReLu) activation [4, 12], with layer sizes 200 and 1000 respectively, before both being mapped to second, combined hidden ReLu layer of dimensionality 1200.”
Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies (2015, background): “Rectifiers are becoming increasingly popular due to their superior empirical performance (Nair and Hinton, 2010; Glorot et al., 2011; Zeiler et al., 2013; Dahl et al., 2013).”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “Currently the most popular activation function for neural networks is the rectified linear unit (ReLU), which was first proposed for restricted Boltzmann machines (Nair & Hinton, 2010) and then successfully used for neural networks (Glorot et al., 2011).”
Scalable agent alignment via reward modeling: a research direction (2018, methodology): “…require non-standard inductive biases; for instance modern deep networks typically use piece-wise linear activation functions (Nair & Hinton, 2010; Glorot et al., 2011; Goodfellow et al., 2013; Xu et al., 2015), which generalize linearly far from training data (Goodfellow et al., 2015), meaning…”
Gradient-Based Neural DAG Learning (2019, neutral): “φ (except at points of non-differentiability arising from the absolute value function, similar to standard neural networks with ReLU activations (Glorot et al., 2011); these points did not appear problematic in our experiments using SGD).”
Knowledge Neurons in Pretrained Transformers (2021, background): “f denotes an activation function such as ReLU (Glorot et al., 2011) or GELU (Hendrycks and Gimpel, 2016).”