Neural Machine Translation with Byte-Level Subwords
By Changhan Wang, Kyunghyun Cho, Jiatao Gu. 2019
220 citations recorded by the source corpus.
Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese however can unnecessarily take up vocabulary slots and limit its compactness. Representing text at the level of bytes and using the 256 byte set as vocabulary is a potential solution to this issue. High computational cost has however prevented it from being widely deployed or used in practice. In this paper, we investigate byte-level subwords, specifically byte-level BPE (BBPE), which is compacter than character vocabulary and has no out-of-vocabulary tokens, but is more efficient than using pure bytes only is. We claim that contextualizing BBPE embeddings is necessary, which can be implemented by a convolutional or recurrent layer. Our experiments show that BBPE has comparable performance to BPE while its size is only 1/8 of that for BPE. In the multilingual setting, BBPE maximizes vocabulary sharing across many languages and achieves better translation quality. Moreover, we show that BBPE enables transferring models between languages with non-overlapping character sets.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Yejin Choi
- Mohit Bansal
- Jiatao Gu
- Wei-Ning Hsu
- Jianfeng Gao
- Graham Neubig
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Yejin Choi
- Luc Van Gool
- Luke S. Zettlemoyer
- Dacheng Tao
- Trevor Darrell
- Jiatao Gu
- Jitendra Malik
- Ming-Hsuan Yang
In Natural Language Processing
- Yejin Choi
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Omer Levy
- Noah A. Smith
- Jiatao Gu
- Mohit Bansal
- Graham Neubig
In Machine Translation
Citations in the corpus
- Direct Speech-to-Speech Translation With Discrete Units (2021, methodology): “MT As the input to the MT model is characters, we follow the default gru_transformer setup in FAIRSEQ (Ott et al., 2019) to prepend a bidirectional recurrent layer with gated recurrent units (GRU) to the transformer encoder to incorporate a larger context (Wang et al., 2020a).”
- Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right (2021, background): “Length normalization is often measured over Byte Pair Encoding (BPE) tokens (Sennrich et al., 2016) and BPE tends to produce vocabularies where most tokens are equally frequent (Wang et al., 2020).”
- Qwen3 Technical Report (2025, unknown): “Qwen3 models utilize Qwen’s tokenizer (Bai et al., 2023), which implements byte-level byte-pair encoding (BBPE, Brown et al., 2020; Wang et al., 2020; Sennrich et al., 2016) with a vocabulary size of 151,669.”
- Neural Machine Translation without Embeddings (2020, methodology): “SentencePiece (Kudo and Richardson, 2018) takes a raw unicode string and tokenizes it into subwords using BPE (Sennrich et al., 2016) or unigram LM (Kudo, 2018).”
- Vocabulary Learning via Optimal Transport for Neural Machine Translation (2020, unknown): “2018; Al-Rfou et al., 2019; Wang et al., 2020).”
- Less training, more repairing please: revisiting automated program repair via zero-shot learning (2022, methodology): “We use the CodeBERT tokenizer which is built using byte-level byte pair encoding (BBPE) – a technique to reduce the size of the vocabulary by breaking uncommon long words into subwords that are found commonly in the corpus [71].”
- Training Multilingual Pre-trained Language Model with Byte-level Subwords (2021, methodology): “It is worthy mentioning that this is a commonly used technique in Neural Machine Translation [21, 4] which we also deploy in this report.”