A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation
By Shuoyang Ding, Adithya Renduchintala, Kevin Duh. 2019
73 citations recorded by the source corpus.
Most neural machine translation systems are built upon subword units extracted by methods such as Byte-Pair Encoding (BPE) or wordpiece. However, the choice of number of merge operations is generally made by following existing recipes. In this paper, we conduct a systematic exploration of different BPE merge operations to understand how it interacts with the model architecture, the strategy to build vocabularies and the language pair. Our exploration could provide guidance for selecting proper BPE configurations in the future. Most prominently: we show that for LSTM-based architectures, it is necessary to experiment with a wide range of different BPE operations as there is no typical optimal BPE configuration, whereas for Transformer architectures, smaller BPE size tends to be a typically optimal choice. We urge the community to make prudent choices with subword merge operations, as our experiments indicate that a sub-optimal BPE configuration alone could easily reduce the system performance by 3-4 BLEU points.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Bill Byrne
- Danielle Saunders
- Benoît Sagot
- Ivan Provilkov
- Dmitrii Emelianenko
- Samson Tan
- Chenglei Si
- Manan Dey
In Computer Vision and Pattern Recognition
- Danielle Saunders
- Bill Byrne
- Ivan Provilkov
- Dmitrii Emelianenko
- Samson Tan
- Chenglei Si
- Manan Dey
- Arun Raja
In Machine Translation
- Ivan Provilkov
- Dmitrii Emelianenko
- K. Jung
- Stéphane Clinchant
- Colin Raffel
- Vassilina Nikoulina
- Bill Byrne
- Danielle Saunders
Citations in the corpus
- BPE-Dropout: Simple and Effective Subword Regularization (2019, background): “Ding et al. (2019) explore the effect of BPE vocabulary size and find that it is better to use small vocabulary for low-resource setting and large vocabulary for a high-resource setting.”
- Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem (2020, positive): “The Hebrew vocabulary is a 2k-merge BPE vocabulary, following the recommendations of Ding et al. (2019) for smaller vocabularies when translating into lower-resource languages.”
- Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, background): “27 A similar study undertaken by Ding et al. (2019) reiterate how contradictory suggestions for number of merges in past work are and add that in low-resource scenarios far fewer merges seem to be better, a trend with Transformers which differs from that with LSTMs, leading to an interesting…”
- On the use of BERT for Neural Machine Translation (2019, methodology): “It has been shown by Ding et al. (2019), these are not optimal settings for IWSLT training because it leads to too many parameters for the amount of data available.”
- Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speech (2022, unknown): “So what if we pre-train the phoneme encoder with only the phoneme as the input? We note that the size of the phoneme dictionary is only around 200, and directly using such a small dictionary would not convey contextual semantic information effectively [24].”
- Vocabulary Learning via Optimal Transport for Neural Machine Translation (2020, unknown): “For example, 30K-40K is the most popular size setting in all 42 papers of Conference of Machine Translation (WMT) through 2017 and 2018 (Ding et al., 2019).”