Multilingual Neural Machine Translation with Language Clustering

By Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao Qin, Tie-Yan Liu. 2019

124 citations recorded by the source corpus.

Multilingual neural machine translation (NMT), which translates multiple languages using a single model, is of great practical importance due to its advantages in simplifying the training process, reducing online maintenance costs, and enhancing low-resource and zero-shot translation. Given there are thousands of languages in the world and some of them are very different, it is extremely burdensome to handle them all in a single model or use a separate model for each language pair. Therefore, given a fixed resource budget, e.g., the number of models, how to determine which languages should be supported by one model is critical to multilingual NMT, which, unfortunately, has been ignored by previous work. In this work, we develop a framework that clusters languages into different groups and trains one multilingual model for each cluster. We study two methods for language clustering: (1) using prior knowledge, where we cluster languages according to language family, and (2) using language embedding, in which we represent each language by an embedding vector and cluster them in the embedding space. In particular, we obtain the embedding vectors of all the languages by training a universal neural machine translation model. Our experiments on 23 languages show that the first clustering method is simple and easy to understand but leading to suboptimal translation accuracy, while the second method sufficiently captures the relationship among languages well and improves the translation accuracy for almost all the languages over baseline methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Machine Translation

Citations in the corpus

Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters (2020, background): “Similar to [21], these learned clusters can be used for training multi-head experiments instead of choosing the clusters manually.”
A Survey on Neural Speech Synthesis (2021, background): “Besides, language similarity [335] can also be considered when conducting the cross-lingual transfer.”
Investigating Multilingual NMT Representations at Scale (2019, neutral): “Some recent works compare learned language vectors (Östling and Tiedemann, 2016; Tan et al., 2019; Tiedemann, 2018) and find conclusions similar to ours.”
Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation (2020, background): “Previous work observed problems of performance degradation, often on high-resource languages, due to interference and constrained capacity (Johnson et al., 2017; Tan et al., 2019).”
A Study of Multilingual Neural Machine Translation (2019, background): “…Hu 2018); (2) one-to-many/many-to-one/many-to-many translation, translation from one source language to multiple target languages (Dong et al. 2015; Tan et al. 2019b; Tan et al. 2019a), from multiple source languages to one target language (Zoph and Knight 2016; Lee, Cho, and Hof-mann 2017), or…”
Multilingual Translation from Denoising Pre-Training (2021, methodology): “Models tuned models, and the ML50 dataset downloading scripts at https://github.com/pytorch/fairseq/tree/ master/examples/multilingual . for all languages (Ha et al., 2016) have also been explored and extended to incorporate language information (Tan et al., 2019).”
Counter-Interference Adapter for Multilingual Machine Translation (2021, methodology): “Johnson et al. (2017) ﬁrst proposed training a single multilingual model via an additional target language tag, which became the paradigm for training multilingual MT models henceforth (Gu et al., 2018; Tan et al., 2018, 2019; Siddhant et al., 2020).”
A Survey of Multilingual Neural Machine Translation (2019, methodology): “In this approach suggested by Tan et al. (2019), bilingual models are ﬁrst trained for all language pairs involved.”
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages (2021, background): “The first token of the source sentence is a special token indicating the source language (Tan et al., 2019; Tang et al., 2020).”
Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (2021, background): “…multilingual encoders and/or decoders (Vázquez et al., 2019; Escolano et al., 2020), using a fixed mix of LS and shared parameters (Wang et al., 2018), inserting lightweight adapters (Bapna & Firat, 2019) and separately modeling languages for different clusters (Tan et al., 2019), to name a few.”