Learning to Segment Inputs for NMT Favors Character-Level Processing
By Julia Kreutzer, Artem Sokolov. 2018
33 citations recorded by the source corpus.
Most modern neural machine translation (NMT) systems rely on presegmented inputs. Segmentation granularity importantly determines the input and output sequence lengths, hence the modeling depth, and source and target vocabularies, which in turn determine model size, computational costs of softmax normalization, and handling of out-of-vocabulary words. However, the current practice is to use static, heuristic-based segmentations that are fixed before NMT training. This begs the question whether the chosen segmentation is optimal for the translation task. To overcome suboptimal segmentation choices, we present an algorithm for dynamic segmentation, that is trainable end-to-end and driven by the NMT objective. In an evaluation on four translation tasks we found that, given the freedom to navigate between different segmentation levels, the model prefers to operate on (almost) character level, providing support for purely character-level NMT models from a novel angle.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Orhan Firat
- Ankur Bapna
- Graham Neubig
- Yi Tay
- Sebastian Ruder
- Luke S. Zettlemoyer
- Felix Stahlberg
- Melvin Johnson
In Computer Vision and Pattern Recognition
- Orhan Firat
- Ankur Bapna
- Naveen Arivazhagan
- Sebastian Ruder
- Yi Tay
- Melvin Johnson
- Donald Metzler
- Yonghui Wu
In Natural Language Processing
- Orhan Firat
- Ankur Bapna
- Graham Neubig
- Rico Sennrich
- Naveen Arivazhagan
- Philipp Koehn
- Melvin Johnson
- Felix Stahlberg
Citations in the corpus
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, positive): “Our results here agree with existing literature (Cherry et al., 2018; Kreutzer and Sokolov, 2018) suggesting that using smaller sub-word tokens, as is the case for smaller vocabularies, performs better in low resource settings due to improved generalization.”
- BPE-Dropout: Simple and Effective Subword Regularization (2019, background): “Several works study how translation quality depends on a level of granularity of a segmentation (Cherry et al., 2018; Kreutzer and Sokolov, 2018; Ding et al., 2019).”
- Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, background): “Prior work also learned segmentation as part of the model but did not scale very well (Wang et al., 2017; Kreutzer and Sokolov, 2018; Kawakami et al., 2019).”
- Vocabulary Learning via Optimal Transport for Neural Machine Translation (2020, unknown): “More recently, some studies notice this problem and propose some practical solutions (Kreutzer and Sokolov, 2018; Cherry et al., 2018; Chen et al., 2019; Salesky et al., 2020).”
- Adapting Multilingual Neural Machine Translation to Unseen Languages (2019, methodology): “This is consistent with previous work supporting character level segmentation for improving NMT of LRL [13, 14].”