xLSTM: Extended Long Short-Term Memory

By Maximilian Beck, Korbinian Poppel, M. Spanring, Andreas Auer, Oleksandra Prudnikova, Michael K Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter. Neural Information Processing Systems, 2024

644 citations recorded by the source corpus.

In the 1990s, the constant error carousel and gating were introduced as the central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have stood the test of time and contributed to numerous deep learning success stories, in particular they constituted the first Large Language Models (LLMs). However, the advent of the Transformer technology with parallelizable self-attention at its core marked the dawn of a new era, outpacing LSTMs at scale. We now raise a simple question: How far do we get in language modeling when scaling LSTMs to billions of parameters, leveraging the latest techniques from modern LLMs, but mitigating known limitations of LSTMs? Firstly, we introduce exponential gating with appropriate normalization and stabilization techniques. Secondly, we modify the LSTM memory structure, obtaining: (i) sLSTM with a scalar memory, a scalar update, and new memory mixing, (ii) mLSTM that is fully parallelizable with a matrix memory and a covariance update rule. Integrating these LSTM extensions into residual block backbones yields xLSTM blocks that are then residually stacked into xLSTM architectures. Exponential gating and modified memory structures boost xLSTM capabilities to perform favorably when compared to state-of-the-art Transformers and State Space Models, both in performance and scaling.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/e2a6ffba64331989858e5078bfb8277343aa90bd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Boris Ginsburg
2. Samuel Kriman
3. Rameswar Panda
4. Fei Jia
5. Dima Rekesh
6. Bailin Wang
7. Yoon Kim
8. Yikang Shen

Citations in the corpus

• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, background): “Mamba [22] and xLSTM [5] use learnable weighting for the recurrence and can match the quality of Transformers in language modeling at small or medium scale.”
• RULER: What's the Real Context Size of Your Long-Context Language Models? (2024, methodology): “Finally, novel architectures (Gu et al., 2022; Fu et al., 2023a; Poli et al., 2023; Fu et al., 2023b; Sun et al., 2023a; Beck et al., 2024; Sun et al., 2024) such as Mamba (Gu & Dao, 2023) and RWKV (Peng et al., 2023) have also been proposed to efficiently handle long-context input.”
• Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, positive): “Recent linear recurrent models typically incorporate a lightweight depthwise-separable convolution layer after the query/key/value projections [30, 9, 18].”
• Learning to (Learn at Test Time): RNNs with Expressive Hidden States (2024, neutral): “Some more recent examples are RWKV [58, 59], xLSTM [4], and Gated Linear Attention (GLA) [82].”
• Gated Linear Attention Transformers with Hardware-Efficient Training (2023, methodology): “This scalar data-dependent gating is also used in Peng et al. (2021), Sun et al. (2024), and Beck et al. (2024).”
• Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, neutral): “This recurrence structure appears in several other architectures including Gated RFA (Peng et al., 2021), xLSTM (Beck et al., 2024), and Gated RetNet (Sun et al., 2024b).”