Byte Pair Encoding is Suboptimal for Language Model Pretraining

By Kaj Bostrom, Greg Durrett. 2020

317 citations recorded by the source corpus.

The success of pretrained transformer language models (LMs) in natural language processing has led to a wide range of pretraining setups. In particular, these models employ a variety of subword tokenization methods, most notably byte-pair encoding (BPE) (Sennrich et al., 2016; Gage, 1994), the WordPiece method (Schuster and Nakajima, 2012), and unigram language modeling (Kudo, 2018), to segment text. However, to the best of our knowledge, the literature does not contain a direct evaluation of the impact of tokenization on language model pretraining. We analyze differences between BPE and unigram LM tokenization, finding that the latter method recovers subword units that align more closely with morphology and avoids problems stemming from BPE’s greedy construction procedure. We then compare the fine-tuned task performance of identical transformer masked language models pretrained with these tokenizations. Across downstream tasks and two languages (English and Japanese), we find that the unigram LM tokenization method matches or outperforms BPE. We hope that developers of future pretrained LMs will consider adopting the unigram LM method over the more prevalent BPE.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, methodology): “For a given FLOPs budget, this allows us to train a tokenizer-free language model that matches the performance of a baseline model trained with a SentencePiece tokenizer, hence removing the need for hand-crafted and potentially harmful tokenization schemes [6, 14].”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, background): “Prior work has highlighted issues with some of these algorithms (Bostrom and Durrett, 2020) and has generally observed that models learned with such rigid tokenization do not cope well with variation in language (Sun et al., 2020).”
An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks (2020, background): “Among them is Bostrom and Durrett (2020), which compared the ﬁne-tuning task performance of BERT (Devlin et al., 2019) pre-trained with BPE and unigram language modeling.”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, background): “Bostrom and Durrett (2020) claim that Uni-gramLM obtains better segmentation than BPE, both qualitatively (they tend to better correspond to morphemes and Morfessor (Creutz and Lagus, 2007) morphs) and quantitatively (they improve BERT-style models’ performance on modern NLP tasks a little in…”
BloombergGPT: A Large Language Model for Finance (2023, background): “, 2016), based on promising results in Kudo and Richardson (2018) and Bostrom and Durrett (2020). Following GPT-2 Radford et al. (2019), we treat our data as a sequence of bytes rather than Unicode characters, and we include each of the 256 bytes as tokens.”
Multimodal Large Language Models: A Survey (2023, methodology): “As a result, subword tokenization methods, such as byte pair encoding [28], divide words into smaller units.”
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, background): “Subword tokenization performs poorly on languages with reduplication (Vania and Lopez, 2017), while byte pair encoding does not align well with morphology (Bostrom and Durrett, 2020).”
Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation (2021, background): “Wang (Kudo, 2018), which builds its vocabulary top-down, was shown to align with morphology better than BPE on pre-trained encoders (Bostrom and Durrett, 2020).”
Speech Recognition via Large-Scale Weak Supervision (unknown): “Durrett, 2020) and, in natural conversation, the average word lasts about 250 mil-liseconds (Yuan et al., 2006), or 25 frames of 10ms.”