BPE-Dropout: Simple and Effective Subword Regularization

By Ivan Provilkov, Dmitrii Emelianenko, Elena Voita. 2019

344 citations recorded by the source corpus.

Subword segmentation is widely used to address the open vocabulary problem in machine translation. The dominant approach to subword segmentation is Byte Pair Encoding (BPE), which keeps the most frequent words intact while splitting the rare ones into multiple tokens. While multiple segmentations are possible even with the same vocabulary, BPE splits words into unique sequences; this may prevent a model from better learning the compositionality of words and being robust to segmentation errors. So far, the only way to overcome this BPE imperfection, its deterministic nature, was to create another subword segmentation algorithm (Kudo, 2018). In contrast, we show that BPE itself incorporates the ability to produce multiple segmentations of the same word. We introduce BPE-dropout - simple and effective subword regularization method based on and compatible with conventional BPE. It stochastically corrupts the segmentation procedure of BPE, which leads to producing multiple segmentations within the same fixed BPE framework. Using BPE-dropout during training and the standard BPE during inference improves translation quality up to 2.3 BLEU compared to BPE and up to 0.9 BLEU compared to the previous subword regularization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Zero-Shot Text-to-Image Generation (2021, methodology): “(5)During training, we apply 10% BPE dropout (Provilkov et al., 2019), whose use is common in the neural machine translation literature.”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, background): “, 2016), and BPE Dropout (Provilkov et al., 2019) for regularization.”
Efficient Training of Language Models to Fill in the Middle (2022, methodology): “Here, a token refers to a word in the byte-pair encoding (BPE) vocabulary.”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, methodology): “To make a model more robust to morphological and compositional generalization, probabilistic segmentation algorithms such as subword regularization (Kudo, 2018) and BPE-dropout (Provilkov et al., 2020) have been proposed, which sample different segmentations during training.”
Formalizing and Benchmarking Prompt Injection Attacks and Defenses (2023, methodology): “Following previous work [25], we use BPE-dropout [39] to re-tokenize data, which maintains the text words with high frequencies intact while breaking the rare ones into multiple tokens.”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, background): “BPE-dropout (Provilkov et al., 2020) is an extension allowing for subword regularization (see §6.4.3.”
Baseline Defenses for Adversarial Attacks Against Aligned Language Models (2023, methodology): “BPE-dropout drops a random p% of the BPE merges during the tokenization of the text, resulting in a randomized tokenization with more tokens than a standard representation.”
The STC System for the CHiME-6 Challenge (2020, positive): “BPEDropout [33] also provided an additional improvement.”
Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation (2021, background): “Subword regularization (Kudo, 2018) and BPE-dropout (Provilkov et al., 2020) recognize that deterministic segmentation during training limits the ability to leverage morphology and word composition; instead, they sample at random one of the multiple tokenizations of the training input, made possible”
Vocabulary Learning via Optimal Transport for Neural Machine Translation (2020, methodology): “To address this problem, we borrow the concept of Marginal Utility in economics (Samuelson, 1937) and propose to use Marginal Utility of Vocabularization (MUV) as the optimization objective.”