Three Strategies to Improve One-to-Many Multilingual Translation

By Yining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu, Chengqing Zong. 2018

78 citations recorded by the source corpus.

Due to the benefits of model compactness, multilingual translation (including many-to-one, many-to-many and one-to-many) based on a universal encoder-decoder architecture attracts more and more attention. However, previous studies show that one-to-many translation based on this framework cannot perform on par with the individually trained models. In this work, we introduce three strategies to improve one-to-many multilingual translation by balancing the shared and unique features. Within the architecture of one decoder for all target languages, we first exploit the use of unique initial states for different target languages. Then, we employ language-dependent positional embeddings. Finally and especially, we propose to divide the hidden cells of the decoder into shared and language-dependent ones. The extensive experiments demonstrate that our proposed methods can obtain remarkable improvements over the strong baselines. Moreover, our strategies can achieve comparable or even better performance than the individually trained translation models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, background): “…the model has a limited capacity while it needs to handle this large number of translation tasks, and sharing all parameters be-tween the different languages can be sub-optimal (Wang et al., 2018) especially if they are not from the same typological language family (Sachan and Neubig, 2018).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Some studies also explore partial parameter sharing strategies tailored for specific architectures (Sachan and Neubig, 2018; Wang et al., 2018b).”
Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (2021, methodology): “…decoupling multilingual encoders and/or decoders (Vázquez et al., 2019; Escolano et al., 2020), using a ﬁxed mix of LS and shared parameters (Wang et al., 2018), inserting lightweight adapters (Bapna & Firat, 2019) and separately modeling languages for different clusters (Tan et al., 2019),…”
Balancing Training for Multilingual Neural Machine Translation (2020, background): “(Zaremoodi et al., 2018; Wang et al., 2018, 2019a) propose improvements to the model architecture to improve multilingual performance, while MultiDDS is a model-agnostic and optimizes multilingual data usage.”
A Survey of Multilingual Neural Machine Translation (2019, background): “[156] explored multiple methods for supporting multiple target languages: (a) target language tag at beginning of the decoder, (b) target language dependent positional embeddings, and (c) divide hidden units of each decoder layer into shared and language-dependent ones.”
Using natural language prompts for machine translation (2022, unknown): “…using language embeddings that we add directly to the intermediate representations of the decoder (Fi-rat et al., 2016) or subword embeddings (Wang et al., 2018; Conneau et al., 2019), task-speciﬁc attention (Blackwood et al., 2018), language tags on the decoder side (Liu et al., 2020), etc.”
Neural Machine Translation for Low-resource Languages: A Survey (2021, methodology): “The common practice is to add a language identiication tag to the source sentence [19, 73, 91, 193].”
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021, background): “Strategies involve allocating language specific hidden states, attention modules, decoders or additional specialized layers ((Hokamp et al., 2019; Wang et al., 2018; Gu et al., 2018; Bapna et al., 2019)).”