ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models

By Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel. 2021

725 citations recorded by the source corpus.

Most widely used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison, token-free models that operate directly on raw text (bytes or characters) have many benefits: They can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text preprocessing pipelines. Because byte or character sequences are longer than token sequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of operating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with minimal modifications to process byte sequences. We characterize the trade-offs in terms of parameter count, training FLOPs, and inference speed, and show that byte-level models are competitive with their token-level counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of pre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our experiments.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “In comparison, mT5 (Xue et al., 2021b) and ByT5 (Xue et al., 2021a) were trained on 6x and 1.5x as much non-English text as PaLM, respectively.”
Scaling Up Models and Data with t5x and seqio (2022, background): “• Scalable T5 • mT5 (Xue et al., 2021) • ByT5 (Xue et al., 2022) These model implementations are minimal in the sense that they only use Flax with limited abstractions as opposed to using higher-level libraries built on top of Flax (e.g. Flaxformer).”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “To the best of our knowledge, GPT-NeoX-20B is the largest dense language model to ever be publicly released with a several-way tie for second place at 13 billion parameters (Artetxe et al., 2021; Xue et al., 2020, 2021) and many more models at the 10-11B parameter scale.”
Scaling Instruction-Finetuned Language Models (2022, background): “9% on one-shot TyDiQA, it is still not on par with ByT5 finetuned on the TyDiQA training set (Xue et al., 2022).”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “[95] trains encoder-decoder T5 models on UTF-8 bytes directly and shows that making the encoder 3x deeper than the decoder leads to comparable performance with subword baselines.”
UL2: Unifying Language Learning Paradigms (2022, methodology): “Pre-training (or denoising) is generally applied on the subword level (Raffel et al., 2019; Devlin et al., 2018) but it is worth to note that it has also been applied on the character or byte-level (Xue et al., 2021; Tay et al., 2021c).”
The Efficiency Misnomer (2021, background): “Token-Free Models Token-free models (Xue et al., 2021; Tay et al., 2021c; Jaegle et al., 2021) get rid of the large subword vocabulary by modeling at the character or byte level.”
SoundStorm: Efficient Parallel Audio Generation (2023, positive): “To model the text-to-semantic token mapping, we train a ByT5-large Transformer (Xue et al., 2022).”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, methodology): “Similar to Xue et al. (2021), we hypothesize that character-level models require deeper encoder stacks than subword-based models to make up for their smaller embedding capacity.”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, methodology): “Most other work uses either words, sub-words (Radford et al., 2018; Devlin et al., 2019), characters (Tay et al., 2021) or even bytes (Xue et al., 2021).”
LeanDojo: Theorem Proving with Retrieval-Augmented Language Models (2023, methodology): “Our backbone model, ByT5 [44], was published in 2021 and has 299M parameters, which is not very large by today’s standard.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Charformer performs on par with models like T5 while outperforming byte-level models like ByT5.”
Aligning Text-to-Image Models using Human Feedback (2023, neutral): “Several techniques, such as character-aware text encoders (Xue et al., 2022; Liu et al., 2022b) and structured representations of language inputs (Feng et al., 2022) have been investigated to address these issues.”
Confident Adaptive Language Modeling (2022, background): “This is especially pronounced in the autoregressive decoding process where the full stack of Transformer layers is repeatedly computed for each output token [37; 40; 86].”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, background): “As shown recently by ByT5 (Xue et al., 2021), byte-level models confer similar beneﬁts to character-level models (better robustness to noise, avoidance of out-of-vocabulary issues, etc.) with similar drawbacks (longer training and inference time due to longer sequences).”