Parameter Sharing Methods for Multilingual Self-Attentional Translation Models

By Devendra Singh Sachan, Graham Neubig. 2018

118 citations recorded by the source corpus.

In multilingual neural machine translation, it has been shown that sharing a single translation model between multiple languages can achieve competitive performance, sometimes even leading to performance gains over bilingually trained models. However, these improvements are not uniform; often multilingual parameter sharing results in a decrease in accuracy due to translation models not being able to accommodate different languages in their limited parameter space. In this work, we examine parameter sharing techniques that strike a happy medium between full sharing and individual training, specifically focusing on the self-attentional Transformer model. We find that the full parameter sharing approach leads to increases in BLEU scores mainly when the target languages are from a similar language family. However, even in the case where target languages are from different families where full parameter sharing leads to a noticeable drop in BLEU scores, our proposed methods for partial sharing of parameters can lead to substantial improvements in translation accuracy.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

1. Graham Neubig
2. Orhan Firat
3. Ankur Bapna
4. Felix Stahlberg
5. Jiatao Gu
6. Tao Qin
7. Philipp Koehn
8. Tie-Yan Liu

Citations in the corpus

• Massively Multilingual Neural Machine Translation (2019, background): “the context of multilingual models (Lakew et al., 2018; Sachan and Neubig, 2018).”
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Some studies also explore partial parameter sharing strategies tailored for specific architectures (Sachan and Neubig, 2018; Wang et al., 2018b).”
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation (2020, neutral): “isting work (Firat et al., 2016b; Sachan and Neubig, 2018), LALT reaches a better trade-off between expressivity and scalability.”
• A Study of Multilingual Neural Machine Translation (2019, background): “(Zoph et al. 2016; Sachan and Neubig 2018) have also studied the sharing of different components of a multilingual model.”
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (2020, methodology): “While prior work have studied the effect of data (Arivazhagan et al., 2019; Wang et al., 2020a), architecture (Blackwood et al., 2018; Sachan & Neubig, 2018; V ´ azquez et al., 2019; Escolano et al., 2020) and scale (Huang et al., 2019b; Lepikhin et al., 2020) on multilingual models, their…”
• Multilingual Translation from Denoising Pre-Training (2021, background): “Subsequently, work focused on the the model capacity bottleneck, leading to exploration of various parameter sharing strategies (Blackwood et al., 2018; Platanios et al., 2018; Sachan and Neubig, 2018; Lu et al., 2018).”
• Importance-based Neuron Allocation for Multilingual Neural Machine Translation (2021, background): “, 2021), specific attention networks (Blackwood et al., 2018; Sachan and Neubig, 2018), adding task adapters (Bapna and Firat, 2019), and training model with different language clusters (Tan et al.”
• One Model, Many Languages: Meta-learning for Multilingual Text-to-Speech (2020, background): “So far, several works explored training joint multilingual models in text-to-speech, following similar experiments in the field of neural machine translation [10, 8].”
• A Survey of Multilingual Neural Machine Translation (2019, methodology): “Sharing encoders among multiple languages is very effective and is widely used (Lee et al., 2017; Sachan and Neubig, 2018).”
• Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “A more efective model allows partial sharing of parameters across decoders [160, 194].”
• Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Models (2019, background): “• partially sharing some of the decoder parameters while keeping others unique to each target language (Sachan and Neubig, 2018; Blackwood et al., 2018).”
• compare-mt: A Tool for Holistic Comparison of Language Generation Systems (2019, background): “4 of Sachan and Neubig (2018) shows the results of frequency-based word accuracy analysis.”