Neural Machine Translation of Rare Words with Subword Units
By Rico Sennrich, Barry Haddow, Alexandra Birch. 2015
8,868 citations recorded by the source corpus.
Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character n-gram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English-German and English-Russian by 1.1 and 1.3 BLEU, respectively.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Graham Neubig
- Percy Liang
- Yoshua Bengio
- Kyunghyun Cho
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Natural Language Processing
- Graham Neubig
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Kyunghyun Cho
- Yejin Choi
- Yonatan Belinkov
- Noah A. Smith
Citations in the corpus
- Attention is All you Need (2017, background): “In terms of computational complexity, self-attention layers are faster than recurrent layers when the sequence length n is smaller than the representation dimensionality d, which is most often the case with sentence representations used by state-of-the-art models in machine translations, such as wor”
- Convolutional Sequence to Sequence Learning (2017, methodology): “In machine translation, this architecture has been demonstrated to outperform traditional phrase-based models by large margins (Sennrich et al., 2016b; Zhou et al., 2016; Wu et al., 2016; § 2).”
- Massive Exploration of Neural Machine Translation Architectures (2017, methodology): “While some work (Luong et al., 2015b; Zhou et al., 2016; Luong and Manning, 2016; Wu et al., 2016) has achieved state of the art results using deep networks, others (Jean et al., 2015; Chung et al., 2016; Sennrich et al., 2016b) have achieved similar results with far shallower ones.”
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “…by training a “copy model” to mimic a traditional alignment model [31], or by using the attention mechanism to copy rare words [37], these approaches are both unreliable at scale, since the quality of the alignments varies across languages, and the latent alignments produced by the attention…”
- Using the Output Embedding to Improve Language Models (2016, methodology): “State of the art results in NMT have recently been achieved by systems that segment the source and target words into subword units (Sennrich et al., 2016a).”
- An efficient framework for learning sentence representations (2018, methodology): “This further limits the size of the vocabulary and the model (Variations of the softmax layer such as hierarchical softmax (Mnih & Hinton, 2009), sampling based softmax (Jean et al., 2014) and sub-word representations (Sennrich et al., 2015) can help alleviate this issue).”
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “We use SentencePiece [Kudo and Richardson, 2018] to encode text as WordPiece tokens [Sennrich et al., 2015; Kudo, 2018].”
- RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, background): “Byte-Pair Encoding (BPE) (Sennrich et al., 2016) is a hybrid between character-and word-level representations that allows handling the large vocabularies common in natural language corpora.”
- Scaling Laws for Neural Language Models (2020, methodology): “We train language models on WebText2, an extended version of the WebText [RWC + 19] dataset, tokenized using byte-pair encoding [SHB15] with a vocabulary size n vocab = 50257 .”
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We tokenize the data with the byte-pair encoding (BPE) algorithm (Sennrich et al., 2015), using the implementation from Sentence-Piece (Kudo and Richardson, 2018).”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “Weuse the same tokenizer asLlama 1; it employs a bytepair encoding (BPE) algorithm (Sennrich et al., 2016) using the implementation from SentencePiece (Kudo and Richardson, 2018).”
- Learning Transferable Visual Models From Natural Language Supervision (2021, background): “The transformer operates on a lower-cased byte pair encoding (BPE) representation of the text with a 49,152 vocab size (Sennrich et al., 2015).”
- Adversarial Training for Large Neural Language Models (2020, methodology): “To address the problem of out-of-vocabulary words, tokens are divided into subword units, using Byte-Pair Encoding (BPE) (Sennrich et al., 2015) or its variants (Kudo and Richardson, 2018), which generates a fixed-size subword vocabulary to compactly represent words in training text corpora.”
- Automatic Detection of Generated Text is Easiest when Humans are Fooled (2019, methodology): “Bag-of-Words For each sequence, we compute a bag-of-words embedding where each dimension corresponds to a token in GPT-2’s 50,000 token BPE vocabulary (Sennrich et al., 2016), and we count how many times that token appears in the text sequence.”
- SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “Text Tokenization We used a data-driven sub-word tokenizer (Sennrich et al., 2016) with a vocabulary size of 2 15 to convert text into integer ID tokens, followed by a 128-dim embedding layer.”
- Technical report on Conversational Question Answering (2019, methodology): “All the texts are tokenized using Byte-Pair Encoding(BPE) (Sennrich et al., 2015) and are chopped to sequences no longer than 512 to-kens.”
- Fine-Tuning Language Models from Human Preferences (2019, methodology): “(2019) trained on their WebText dataset and their 50,257 token invertible byte pair encoding to preserve capitalization and punctuation (Sennrich et al., 2015).”
- MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “For each task, we jointly learn a 60,000 sub-word units with Byte-Pair Encoding (Sennrich et al., 2016) between source and target languages.”
- Massively Multilingual Neural Machine Translation (2019, methodology): “Since the dataset is already tokenized we did not apply additional preprocessing other than applying joint sub-word segmentation (Sennrich et al., 2016) with 32k symbols.”
- Cross-lingual Language Model Pretraining (2019, methodology): “In all our experiments we process all languages with the same shared vocabulary created through Byte Pair Encoding (BPE) (Sennrich et al., 2015).”