Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units

By Dan Hendrycks, Kevin Gimpel. 2016

819 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “The ﬁrst part of the block is a GEMM followed by a GeLU nonlinearity: One option to parallelize the GEMM is to split the weight matrix A along its rows and input X along its columns as: This partitioning will result in ) and this approach will require a synchronization point before the GeLU function”
Generating Long Sequences with Sparse Transformers (2019, background): “Our choice of f is the Gaussian Error Linear Unit (Hendrycks & Gimpel, 2016), f ( X ) = X (cid:12) sigmoid(1 .”
Cross-lingual Language Model Pretraining (2019, methodology): “In all experiments, we use a Transformer architecture with 1024 hidden units, 8 heads, GELU activations (Hendrycks and Gimpel, 2016), a dropout rate of 0.1 and learned positional embed-dings.”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, positive): “In the non-MoE Transformer feed-forward sub-layers, we replace the first linear projection and the activation function with the Gated Linear Unit (Dauphin et al., 2017; Shazeer, 2020), which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error L”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, methodology): “We replace ReLU with GeLU (Hendrycks and Gimpel, 2016) as activation function on feed forward network.”
GLU Variants Improve Transformer (2020, methodology): “…[Raﬀel et al., 2019] 1 , we use a version with no bias: Subsequent work has proposed replacing the ReLU with other nonlinear activation functions such as Gaussian Error Linear Units, GELU( x ) = x Φ( x ) [Hendrycks and Gimpel, 2016], and Swish β ( x ) = xσ ( βx ) [Ramachandran et al., 2017].”
Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, methodology): “PFFN(·) is Positionwise Feed-forward Network applied to all positions separately and identically: PFFN(x) = W2GELU(W1x + b1) + b2, where GELU(·) is the Gaussian Error Linear Unit (GELU) activation function [13].”
LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, methodology): “To use our LXMERT model on NLVR 2 , we concatenate the cross-modality representations of the two images and then build the classiﬁer with GeLU activation(Hendrycks and Gimpel, 2016).”
A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning (2019, background): “Here, hCLS is the first vector in the final contextualized representation M3, and FFN denotes a feed-forward network consisting of two linear projections with a GeLU activation (Hendrycks and Gimpel, 2016) followed by a layer normalization (Lei Ba et al.”
Fast Inference from Transformers via Speculative Decoding (2022, methodology): “The model here is a GPT-like Transformer decoder with Gelu activations (Hendrycks & Gimpel, 2016).”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “• Transformer+GELU: A commonly used variant of the vanilla Transformer that uses a GELU [40] approximation activation function [2, 7].”
A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks (2016, methodology): “We initialize our weights according to (Hendrycks & Gimpel, 2016c), as it is suited for arbitrary nonlinearities.”
COMET: Commonsense Transformers for Automatic Knowledge Graph Construction (2019, methodology): “We use a dropout rate of 0.1 and use GeLU (Hendrycks and Gimpel, 2016) units as activation functions.”
Early Methods for Detecting Adversarial Images (2016, methodology): “We Distance from mean KL-Div use a hidden layer size of 256, the Adam Optimizer (Kingma & Ba, 2015) with suggested parameters, and the GELU nonlinearity (Hendrycks & Gimpel, 2016).”
Searching for MobileNetV3 (2019, methodology): “In [36, 13, 16] a nonlinearity called swish was introduced that when used as a drop-in replacement for ReLU, that significantly improves the accuracy of neural networks.”