Hyena Hierarchy: Towards Larger Convolutional Language Models

By Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen A. Baccus, Yoshua Bengio, Stefano Ermon, Christopher R'e. International Conference on Machine Learning, 2023

532 citations recorded by the source corpus.

Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale. However, the core building block of Transformers, the attention operator, exhibits quadratic cost in sequence length, limiting the amount of context accessible. Existing subquadratic methods based on low-rank and sparse approximations need to be combined with dense attention layers to match Transformers, indicating a gap in capability. In this work, we propose Hyena, a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. In recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens, Hyena improves accuracy by more than 50 points over operators relying on state-spaces and other implicit and explicit methods, matching attention-based models. We set a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile), reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K. Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100x faster at sequence length 64K.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/998ac3e945857cf2676ee7efdbaf443a0c6f820a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Retentive Network: A Successor to Transformer for Large Language Models (2023, methodology): “The third line explores replacing attention with other mechanisms, such as S4 (Gu et al., 2021), and its variants (Dao et al., 2022b; Poli et al., 2023).”
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution (2023, methodology): “The HyenaDNA model is a decoder-only, sequence-to-sequence architecture defined by a stack of blocks consisting of a Hyena operator [37], followed by a feed-forward neural network (see Fig.”
Lost in the Middle: How Language Models Use Long Contexts (2023, background): “Finally, a variety of recently-proposed architectures model sequences with millions of tokens, raising the potential of further dramatic increases in language model maximum context length (Gu et al., 2022; Fu et al., 2023; Poli et al., 2023; Yu et al., 2023, inter alia ).”
Rewrite the Stars (2024, unknown): “Star operation exhibits promising performance and efﬁciency across various research ﬁelds, including Natural Language Processing ( i.e. , Monarch Mixer [16], Mamba [17], Hyena Hierarchy [44], 45, 60], with “ ?”
RWKV: Reinventing RNNs for the Transformer Era (2023, background): “State space models (SSM) like S4 (Gu et al., 2022) and its variants (Dao et al., 2022b; Poli et al., 2023) are also proposed.”
Mechanistic Interpretability for AI Safety - A Review (2024, methodology): “This focus on cherry-picked results from toy models is a form of streetlight interpretability (Casper, 2023), examining AI systems under only ideal conditions of maximal interpretability.”
Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, unknown): “For example, this framing makes it difﬁcult to (neatly) capture other subquadratic models that have been shown to be effective [117, 48, 95, 81].”
Large Language Models: A Survey (2024, unknown): “Some recent models in this category are Mamba [30], Hyena [210], and Striped Hyena [211].”
Jamba: A Hybrid Transformer-Mamba Language Model (2024, background): “Closest are perhaps H3 [14], a specially designed SSM that enables induction capabilities, and a generalization called Hyena [35].”
Learning to (Learn at Test Time): RNNs with Expressive Hidden States (2024, unknown): “The general idea of fast weights is to update the parameters of a “fast” model on only the most relevant data, as opposed to the conventional practice of updating a “slow” model on all data [74].”
RULER: What's the Real Context Size of Your Long-Context Language Models? (2024, methodology): “Finally, novel architectures (Gu et al., 2022; Fu et al., 2023a; Poli et al., 2023; Fu et al., 2023b; Sun et al., 2023a; Beck et al., 2024; Sun et al., 2024) such as Mamba (Gu & Dao, 2023) and RWKV (Peng et al., 2023) have also been proposed to efficiently handle long-context input.”
Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT (2024, unknown): “Recent work on state-space architectures, such as S4 [16], Mamba [15], Monarch Mixer [10], and more [12, 19, 35, 43, 47], suggests that the subquadratic scaling properties enjoyed by these models make them amenable for long contexts.”