MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

By L. Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke S. Zettlemoyer, Michael Lewis. 2023

177 citations recorded by the source corpus.

Autoregressive transformers are spectacular models for short sequences but scale poorly to long sequences such as high-resolution images, podcasts, code, or books. We proposed Megabyte, a multi-scale decoder architecture that enables end-to-end differentiable modeling of sequences of over one million bytes. Megabyte segments sequences into patches and uses a local submodel within patches and a global model between patches. This enables sub-quadratic self-attention, much larger feedforward layers for the same compute, and improved parallelism during decoding -- unlocking better performance at reduced cost for both training and generation. Extensive experiments show that Megabyte allows byte-level models to perform competitively with subword models on long context language modeling, achieve state-of-the-art density estimation on ImageNet, and model audio from raw files. Together, these results establish the viability of tokenization-free autoregressive sequence modeling at scale.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Percy Liang
2. Mohit Bansal
3. Jianfeng Gao
4. Gabriel Synnaeve
5. Luke S. Zettlemoyer
6. Yonatan Belinkov
7. Yossi Adi
8. Hannaneh Hajishirzi

In Natural Language Processing

1. Yonatan Belinkov
2. Percy Liang
3. Gabriel Synnaeve
4. Thomas Scialom
5. John Hewitt
6. Luke S. Zettlemoyer
7. Mohit Bansal
8. Christopher D. Manning

In Model Architectures

1. Hugo Touvron
2. Anima Anandkumar
3. Percy Liang
4. John Hewitt
5. Yuan-Jui Chen
6. Jiang Bian
7. Nicolas Usunier
8. Joanna Bitton

In Signal Processing

1. Yossi Adi
2. Gabriel Synnaeve
3. Dongchao Yang
4. Rongjie Huang
5. Jade Copet
6. Tal Remez
7. Jiang Bian
8. Xuankai Chang

Citations in the corpus

• Code Llama: Open Foundation Models for Code (2023, background): “Long sequences in LLMs. Scaling Transformers and LLMs to long input sequences has attracted much recent interest (Dai et al., 2019; Beltagy et al., 2020; Yu et al., 2023; Ding et al., 2023).”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “Inspired by (Yu et al., 2023), a multi-scale Transformer architecture is specifically designed for discrete audio sequences, which is a hierarchical model that processes the inter-and intra-frame correlation by global and local Transformer modules separately.”
• LM-VC: Zero-Shot Voice Conversion via Speech Generation Based on Language Models (2023, background): “With bigger GPU memory or specific-designed structures [38], [39], this problem may be alleviated with the increased training length.”
• Lost in the Middle: How Language Models Use Long Contexts (2023, background): “4 To modulate the position of relevant information within the input context, we adjust the order of the documents to change the position of the document that contains the answer (Figure 3).”
• LeanDojo: Theorem Proving with Retrieval-Augmented Language Models (2023, methodology): “To solve the issue, it might be helpful to pretrain a customized tokenizer or adopt more advanced tokenizer-free models such as MegaByte [105].”