Gated Delta Networks: Improving Mamba2 with Delta Rule

By Songlin Yang, Jan Kautz, Ali Hatamizadeh. International Conference on Learning Representations, 2024

381 citations recorded by the source corpus.

Linear Transformers have gained attention as efficient alternatives to standard Transformers, but their performance in retrieval and long-context tasks has been limited. To address these limitations, recent work has explored two distinct mechanisms: gating for adaptive memory control and the delta update rule for precise memory modifications. We observe that these mechanisms are complementary: gating enables rapid memory erasure while the delta rule facilitates targeted updates. Building on this insight, we introduce the gated delta rule and develop a parallel training algorithm optimized for modern hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses existing models like Mamba2 and DeltaNet across multiple benchmarks, including language modeling, common-sense reasoning, in-context retrieval, length extrapolation, and long-context understanding. We further enhance performance by developing hybrid architectures that combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, achieving both improved training efficiency and superior task performance.

Subfields: Hardware and Architecture

View Paper: https://www.semanticscholar.org/paper/968ea1c7f681592ce6b0830eac0b7fda8db71792

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Hardware and Architecture

1. Bailin Wang
2. Yoon Kim
3. Yikang Shen
4. Songlin Yang

Citations in the corpus

• Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, positive): “Our main baselines include: LLaMA-architecture Transformer++ [111]; RetNet [105], a linear attention Transformer with non-data-dependentexponential decay and large head dimension; GLA [116], a linear attention Transformer with data-dependent decay; and Mamba [30], a selective state-space model with ”
• A Survey of Model Architectures in Information Retrieval (2025, unknown): “Recent advances in linear RNNs (Peng et al., 2023, 2024; Qin et al., 2024b), state space models (Gu and Dao, 2024; Dao and Gu, 2024), and linear attention (Katharopoulos et al., 2020; Yang et al., 2024) offer alternatives with theoretical linear complexity.”