Gaussian Error Linear Units (GELUs)
By Dan Hendrycks, Kevin Gimpel. 2016
7,098 citations recorded by the source corpus.
We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is $x\Phi(x)$, where $\Phi(x)$ the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs ($x\mathbf{1}_{x>0}$). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Yoshua Bengio
- Luke S. Zettlemoyer
- Percy Liang
- Graham Neubig
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Alan L. Yuille
In Model Architectures
- Quoc V. Le
- Jianfeng Gao
- Alan L. Yuille
- Yoshua Bengio
- Jiashi Feng
- Shuicheng Yan
- Christoph Feichtenhofer
- Kaiming He
Citations in the corpus
- RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, positive): “BERT trains with a dropout of 0.1 on all layers and attention weights, and a GELU activation function (Hendrycks and Gimpel, 2016).”
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “We also found the GELU activation function (Hendrycks & Gimpel, 2016) to perform better for this task.”
- Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “We investigate BERT-based models (Devlin et al., 2019) which are pretrained bidirectional Transformers (Vaswani et al., 2017) with GELU (Hendrycks and Gimpel, 2016) activations.”
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, methodology): “BART uses the standard sequence-to-sequence Trans-former architecture from (Vaswani et al., 2017), except, following GPT, that we modify ReLU activation functions to GeLUs (Hendrycks & Gimpel, 2016) and initialise parameters from N (0 , 0 .”
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, methodology): “The backbone of the ALBERT architecture is similar to BERT in that it uses a transformer encoder (Vaswani et al., 2017) with GELU nonlinearities (Hendrycks & Gimpel, 2016).”
- Generating Long Sequences with Sparse Transformers (2019, methodology): “Our choice of f is the Gaussian Error Linear Unit (Hendrycks & Gimpel, 2016), f(X) = X sigmoid(1.702 ·X), as used in (Radford et al., 2018).”
- StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding (2019, methodology): “The gelu activation [22] was used as GPT [8].”
- SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, methodology): “We also keep a dropout of 0.1 on all layers and attention weights, and a GeLU activation function (Hendrycks and Gimpel, 2016).”
- Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, methodology): “The gelu activation (Hendrycks and Gimpel, 2016) was used as GPT (Radford et al., 2018).”
- Galactica: A Large Language Model for Science (2022, methodology): “Galactica uses a Transformer architecture in a decoder-only setup (Vaswani et al., 2017), with the following modifications: • GeLU Activation - we use GeLU activations for all model sizes (Hendrycks and Gimpel, 2016).”
- Mixture-of-Experts with Expert Choice Routing (2022, methodology): “…(only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.”
- PALM: Pre-training an Autoencoding&autoregressive Language Model for Context-conditioned Generation (2020, methodology): “1 on all layers and attention weights, and a GELU activation function (Hendrycks and Gimpel, 2016) used as GPT.”
- VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “We closely follow the transformer decoder architecture from [29], but use GELU [85] rather than ReLU, following [64, 79].”
- Benchmarking Detection Transfer Learning with Vision Transformers (2021, methodology): “The first of these modules upsamples the feature map by a factor of 4 using a stride-two 2 × 2 transposed convolution, followed by group normalization [39] and GeLU [21], and finally another stride-two 2 × 2 transposed convolution.”
- Training data-efficient image transformers & distillation through attention (2020, background): “This FFN is composed of two linear layers separated by a GeLu activation [21].”
- Perceiver: General Perception with Iterative Attention (2021, positive): “In the dense block, inputs are processed with layer norm, passed through a linear layer, activated with a GELU nonlinearity (Hendrycks & Gimpel, 2016), and passed through a final linear layer.”