Balancing Training for Multilingual Neural Machine Translation

By Yulia Tsvetkov, Graham Neubig. 2020

116 citations recorded by the source corpus.

When training multilingual machine translation (MT) models that can translate to/from multiple languages, we are faced with imbalanced training sets: some languages have much more training data than others. Standard practice is to up-sample less resourced languages to increase representation, and the degree of up-sampling has a large effect on the overall performance. In this paper, we propose a method that instead automatically learns how to weight training data through a data scorer that is optimized to maximize performance on all test languages. Experiments on two sets of languages under both one-to-many and many-to-one MT settings show our method not only consistently outperforms heuristic baselines in terms of average performance, but also offers flexible control over the performance of which languages are optimized.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, background): “However, to maintain balance between groups (Wang et al., 2020), we cluster the remaining languages together and roughly balance the amount of training data for each group.”
Complete Multilingual Neural Machine Translation (2020, neutral): “…heuristics (Lee et al., 2016; Devlin et al., 2018; Arivazhagan et al., 2019b), others relying on adaptive schedules that incorporate the model gains, baselines or quality expectations into the data schedulers (Kiper-wasser and Ballesteros, 2018; Jean et al., 2019; Wang et al., 2020).”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “Multilingual MT (Dong et al., 2015; Firat et al., 2016a,b; Wang et al., 2020) addresses the transfer of MT from high-resourced to low-resourced languages by training multilingual models for all languages at once.”
Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2021, methodology): “also sample examples in proportion to the size of the training set (Raffel et al., 2020; Wang et al., 2020).”
Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (2021, background): “…interning at Google Research. transfer across languages, it also introduces task-interference between dissimilar languages (Arivazhagan et al., 2019; Wang et al., 2020a;b) and a capacity bottleneck emerges due to the increased number of languages and data (Huang et al., 2019; Zhang et al., 2020b).”
Scaling End-to-End Models for Large-Scale Multilingual ASR (2021, background): “However, high resource languages where sufficient monolingual data exists, suffer from interference and constrained capacity [19, 24], and often see a degradation in performance.”
Measuring The Impact Of Programming Language Distribution (2023, background): “adopted more dynamic approaches, adapting the sampling rates in an online fashion during training (Wang et al., 2020).”
Ranking Transfer Languages with Pragmatically-Motivated Features for Multilingual Sentiment Analysis (2020, background): “Previous work has shown that selecting the right set of training languages can significantly boost the performance of cross-lingual models (Paul et al., 2009; Lin et al., 2019; Wang and Neubig, 2019; Wang et al., 2020).”
Tricks for Training Sparse Translation Models (2021, background): “Learned data sam-Model plers (Wang et al., 2020) can choose better sampling schemes but are computationally expensive.”
Deep Transformers with Latent Depth (2020, methodology): “Theoretical results suggest networks often have an expressive power that scales exponentially in depth instead of width [21], and recent work [36, 1, 23, 32] ﬁnds that deeper Transformers improve performance on various generation tasks.”
Neural Machine Translation for Low-resource Languages: A Survey (2021, unknown): “These sampling strategies include simple over-sampling and temperature-based sampling [6, 57, 188, 190].”
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (2020, methodology): “…training paradigm (Arivazhagan et al., 2019; Wang et al., 2020b), and how to further improve it by utilizing post-hoc alignment (Wang et al., 2020c; Cao et al., 2020), data balancing (Jean et al., 2019; Wang et al., 2020a), or calibrated training signal (Mulcaire et al., 2019; Huang et al., 2019a).”
Class-Weighted Classification: Trade-offs and Robust Approaches (2020, background): “…the number of hyperparameters to only 𝑐 ∈ (0 , 1] and 𝜅 ∈ (0 , ∞ ), we calculate 𝛼 𝑖 as follows: 𝜅 behaves as a temperature parameter (similar to Jang et al. 2016; Wang et al. 2020) and causes 𝛼 to become a smoother distribution of weights when 𝜅 > 1 and converge to uniform weights as 𝜅 →…”