GLU Variants Improve Transformer

By Noam Shazeer. 2020

2,062 citations recorded by the source corpus.

Gated Linear Units (arXiv:1612.08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers of the Transformer (arXiv:1706.03762) sequence-to-sequence model, and find that some of them yield quality improvements over the typically-used ReLU or GELU activations.

Subfields: Artificial Intelligence, Biomedical Engineering, Hardware and Architecture, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Hardware and Architecture

In Model Architectures

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “• SwiGLU Activation – We use SwiGLU activations (Swish(xW ) · xV ) for the MLP intermediate activations because they have been shown to significantly increase quality compared to standard ReLU, GeLU, or Swish activations (Shazeer, 2020).”
LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We replace the ReLU non-linearity by the SwiGLU activation function, introduced by Shazeer (2020) to improve the performance.”
Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We use the standard transformer architecture (Vaswani et al., 2017), apply pre-normalization using RMSNorm (Zhang and Sennrich, 2019), use the SwiGLU activation function (Shazeer, 2020), and rotary positional embeddings (RoPE, Su et al. 2022).”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, methodology): “…feed-forward sub-layers, we replace the first linear projection and the activation function with the Gated Linear Unit (Dauphin et al., 2017; Shazeer, 2020), which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit…”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “1” recipe,5 which improves upon T5 by using GeGLU nonlinearities (Shazeer, 2020), scaling both dmodel and dff instead”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “For example, alternative positional embedding schemes (Su et al., 2021; Press et al., 2021) or novel activation functions (Shazeer, 2020).”
The Power of Scale for Parameter-Efficient Prompt Tuning (2021, methodology): “These improvements are (1) the removal of all supervised data from pre-training, (2) adjustments to hyperparameters dmodel and dff , and (3) the use of GeGLU (Shazeer, 2020) over ReLU (Nair and Hinton, 2010) activations.”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, methodology): “1.1 architecture (Shazeer, 2020) for all of our experiments — which uses GEGLU-activated layers instead of ReLU in classic Transformer models (Vaswani et al., 2017).”
UL2: Unifying Language Learning Paradigms (2022, methodology): “UL2 adopts a pretty standard vanilla T5 Transformer that have been enhanced with modifications that have withstood the test of time, i.e., GLU layers (Shazeer, 2020) and T5-style relative attention.”
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, background): “Detailed pretraining hyperparameters can be found in Table 2: we based elements of our pretraining setup (such as Adafactor, GEGLU, and the use of an auxiliary Z loss L ( Z ) = 10 − 4 ∗ log 2 ( Z ) to stabilize training [Chowdhery et al., 2022]) on the popular T5.”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “• Transformer++: A Transformer with the following enhancements: RMS normalization [47], Swish activation [48] and a GLU multiplicative branch [38] in the feed forward inverted bottleneck (SwiGLU) [39].”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, background): “We also explore “Gated Linear Unit” (GLU) variants (Dauphin et al., 2017; Shazeer, 2020) which compose two linear transformations together in an element-wise fashion, i.e.”
What Language Model to Train if You Have One Million GPU Hours? (2022, methodology): “We evaluate a recently proposed alternative, SwiGLU (Shazeer, 2020), which combines both Gated Linear Units”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, methodology): “3 For the re-scaled variant, we also used the GLU variant described in (Shazeer, 2020) which is commonly referred to as the V1.”
gpt-oss-120b&gpt-oss-20b Model Card (2025, positive): “The MoE blocks use the gated SwiGLU [9] activation function 1 .”
Jamba: A Hybrid Transformer-Mamba Language Model (2024, background): “Other architecture details are standard, including grouped-query attention (GQA), SwiGLU activation function [6, 40, 45], and load balancing for the MoE [13].”