Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

By Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, Donald Metzler. 2021

213 citations recorded by the source corpus.

State-of-the-art models in natural language processing rely on separate rigid subword tokenization algorithms, which limit their generalization ability and adaptation to new settings. In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. To this end, we introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters in a data-driven fashion. Concretely, GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using a block scoring network. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level. Via extensive experiments on English GLUE, multilingual, and noisy text datasets, we show that Charformer outperforms a series of competitive byte-level baselines while generally performing on par and sometimes outperforming subword-based models. Additionally, Charformer is fast, improving the speed of both vanilla byte-level and subword-level Transformers by 28%-100% while maintaining competitive quality. We believe this work paves the way for highly performant token-free models that are trained completely end-to-end.

Subfields: Artificial Intelligence, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, methodology): “We report all three key4 efficiency metrics (number of params, 3Our concurrent work Charformer (Tay et al., 2021b) makes use of a DeepNarrow inspired strategy which is referred to as Tall in the paper.”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, methodology): “[83] trains on UTF-8 bytes directly by introducing a hand-designed module that is trained end-to-end to perform subword tokenization and produces results on-par with and sometimes better than subword-based models.”
Efficient Transformers: A Survey (2020, methodology): “Even though many existing ‘memory’ models utilize a form of downsampling to achieve a speed and eﬃciency gain, we erected a new categoriation of ‘downsampling’ to better reﬂect this new emerging trend (Dai et al., 2020; Jaegle et al., 2021; Tay et al., 2021c; Ryoo et al., 2021).”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, methodology): “, 2019), characters (Tay et al., 2021) or even bytes (Xue et al.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Charformer performs on par with models like T5 while outperforming byte-level models like ByT5.”
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022, background): “…that the tokenizer plays an important143 role in the per-language downstream task perfor-144 mance, which Clark et al. (2021); Xue et al. (2021);145 Tay et al. (2021) take to the extreme by proposing146 tokenizer-free approaches.147 To extend a monolingual LM to other languages,148 Artetxe et al.…”
Unifying Language Learning Paradigms (2022, methodology): “Pre-training (or denoising) is generally applied on the subword level (Raffel et al., 2019; Devlin et al., 2018) but it is worth to note that it has also been applied on the character or byte-level (Xue et al., 2021; Tay et al., 2021c).”
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution (2023, background): “Consequently, these aggregation techniques create large new vocabularies (compared to the natural vocabulary of 4 nucleotides) that are less generalizable [49].”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, methodology): “, 2021), or Charformer (Tay et al., 2021) make maximally decomposed processing feasible for certain domains and use-cases, they do not cover the wide variety of NLP scenarios and come with their own drawbacks and biases.”
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, background): “The model has both English and multilingual variants and has demonstrated outstanding performance on language understanding tasks, such as the classification of long text documents (Tay et al., 2022).”
Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (2023, methodology): “Recently, however, SotA paraphrase identiﬁcation approaches have primarily used BERT-based models to classify pairs of sequences into the classes paraphrases and not paraphrases (He et al., 2020b; Tay et al., 2021).”
A New Generation of Perspective API: Efficient Multilingual Character-level Transformers (2022, methodology): “We refer interested readers to [29] for fine-grained details.”
Detecting hallucinations in large language models using semantic entropy (2024, unknown): “This builds on past work towards paraphrase identification based on embedding similarity 59,60 and BERT-style models 61,62 .”