Vocabulary Learning via Optimal Transport for Neural Machine Translation

By Jingjing Xu, Hao Zhou, Chun Gan, Zaixiang Zheng, Lei Li. 2020

104 citations recorded by the source corpus.

The choice of token vocabulary affects the performance of machine translation. This paper aims to figure out what is a good vocabulary and whether we can find the optimal vocabulary without trial training. To answer these questions, we first provide an alternative understanding of vocabulary from the perspective of information theory. It motivates us to formulate the quest of vocabularization – finding the best token dictionary with a proper size – as an optimal transport (OT) problem. We propose VOLT, a simple and efficient solution without trial training. Empirical results show that VOLT beats widely-used vocabularies in diverse scenarios, including WMT-14 English-German translation, TED bilingual translation, and TED multilingual translation. For example, VOLT achieves 70% vocabulary size reduction and 0.5 BLEU gain on English-German translation. Also, compared to BPE-search, VOLT reduces the search time from 384 GPU hours to 30 GPU hours on English-German translation. Codes are available at https://github.com/Jingjing-NLP/VOLT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation (2022, methodology): “For TED bilingual dataset, we use the pre-processed data used in Xu et al. (2021)3.”
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, background): “Later works extended optimal transport to the deep learning landscape for applications such as document classification [34], fewshot learning [81], domain adaptation [18], self-supervised learning [46], neural machine translation [76], and comprehending scene using 3D point cloud from monocular data”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, background): “…happens with far fewer merges than are generally required to reach good performance. lem as a search for the set of tokens with the highest entropy, Xu et al. (2021) proposes an optimal transport driven selection from BPE units that obtains vocabulary merges that often outperform a…”
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, background): “Only 1 in 8 papers did; the best paper (Xu et al., 2021), like most two-dimensional ACL 2021 papers, considered multilinguality and efficiency.”