Rectified Linear Units Improve Restricted Boltzmann Machines

By Vinod Nair, Geoffrey E. Hinton. 2010

18,880 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Deep Residual Learning for Image Recognition (2015, background): “ReLU [29] and the biases are omitted for simplifying notations.”
Learned in Translation: Contextualized Word Vectors (2017, methodology): “A function f applies a feedforward network with ReLU activation [Nair and Hinton, 2010] to each element of w̃ and w̃, and a bidirectional LSTM processes the resulting sequences to obtain task specific representations, x = biLSTM(f(w̃)) (7) y = biLSTM(f(w̃)) (8) These sequences are each stacked along”
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (2015, background): “ReLUs [15] are applied to the output of the n × n conv layer.”
Object Detection Networks on Convolutional Feature Maps (2015, background): “We investigate using 1 to 3 additional conv layers (with ReLU) in a NoC.”
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015, methodology): “For example, consider a layer with the input u that adds the learned bias b , and normalizes the result by subtracting the mean of the activation computed over the training data: b − − Thus, the combination of the update to b and subsequent change in normalization led to no change in the output of…”
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, background): “Among these advances, the rectifier neuron [24, 9, 23, 38], e.”
Convolutional neural networks at constrained time cost (2014, methodology): “All the convolutional/fc layers (except the last fc) are with the Rectified Linear Units (ReLU) [18, 14].”
On the Number of Linear Regions of Deep Neural Networks (2014, neutral): “…hand, recently it has become more common to use piecewise linear functions, such as the rectiﬁer activation g ( a ) = max 0 , a (Glorot et al. 2011, Nair and Hinton 2010) or the maxout activation g ( a 1 , . . . 1.0 These previous theoretical results, however, do not trivially apply to the types…”
Mask R-CNN (2017, methodology): “All convs are 3×3, except the output conv which is 1×1, deconvs are 2×2 with stride 2, and we use ReLU [30] in hidden layers.”
Playing Atari with Deep Reinforcement Learning (2013, positive): “The first hidden layer convolves 16 8× 8 filters with stride 4 with the input image and applies a rectifier nonlinearity [10, 18].”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “We then use projected, residual feedforward networks (FFN) with ReLU activations [Nair and Hinton, 2010, Vaswani et al., 2017] and layer normalization [Ba et al., 2016] on the inputs and outputs.”
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation (2013, background): “The prototypical example of that situation is the rectiﬁer unit (Nair and Hinton, 2010; Glorot et al. , 2011), whose non-linearity is simply max(0 , arg) , i.e., h i = max(0 , z i + a i ) where z i is a zero-mean noise.”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “…k heads that uses a causal mask to preclude attending to future tokens: The core of the second block is a feedforward network with ReLU activation (Nair & Hinton, 2010) that projects inputs to an inner dimension f , with parameters U ∈ R d × f and V ∈ R f × d : Each block precedes core…”
Unifying Question Answering and Text Classification via Span Extraction (2019, methodology): “This is followed by a feedforward network with ReLU activation (Nair and Hinton, 2010; Vaswani et al., 2017), another residual connection, and a final layer normalization.”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “Common sequence to sequence learning methods decompose this objective as p(yn, yn−1, ..., y1|x1, x2, ..., xm) =∏n t=1 p(yt|yt−1, ..., y1;x1, x2, ..., xm).”
Gaussian Error Linear Units (GELUs) (2016, neutral): “But as networks became deeper, training with sigmoid activations proved less effective than the non-smooth, less-probabilistic ReLU (Nair & Hinton, 2010) which makes hard gating decisions based upon an input’s sign.”
Adversarial Training Methods for Semi-Supervised Text Classification (2016, background): “The activation function on the hidden layer was ReLU(Jarrett et al., 2009; Nair & Hinton, 2010; Glorot et al., 2011).”
Character-level Convolutional Networks for Text Classification (2015, background): “The non-linearity used in our model is the rectifier or thresholding function h(x) = max{0, x}, which makes our convolutional layers similar to rectified linear units (ReLUs) [24].”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, methodology): “Every expert FFNe applies to xs a fully-connected 2-layer network using ReLU [29] activation function.”
Learning Recurrent Span Representations for Extractive Question Answering (2016, methodology): “Hidden layers in the feed forward neural networks use rectiﬁed linear units (Nair & Hinton, 2010).”