Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoders

By Carlos Escolano, Marta Ruiz Costa-jussà, José A. R. Fonollosa, Mikel Artetxe. 2020

60 citations recorded by the source corpus.

State-of-the-art multilingual machine translation relies on a universal encoder-decoder, which requires retraining the entire system to add new languages. In this paper, we propose an alternative approach that is based on language-specific encoder-decoders, and can thus be more easily extended to new languages by learning their corresponding modules. So as to encourage a common interlingua representation, we simultaneously train the N initial languages. Our experiments show that the proposed approach outperforms the universal encoder-decoder by 3.28 BLEU points on average, while allowing to add new languages without the need to retrain the rest of the modules. All in all, our work closes the gap between shared and language-specific encoderdecoders, advancing toward modular multilingual machine translation systems that can be flexibly extended in lifelong learning settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Gender Bias in Machine Translation (2021, background): “(2020b) demonstrate that architectural choices in multilingual MT impact systems’ behavior: shared encoder-decoders retain less gender information in the source embeddings and less diversion in the attention than language-specific encoder-decoders (Escolano et al., 2021), thus disfavouring the gener”
Importance-based Neuron Allocation for Multilingual Neural Machine Translation (2021, methodology): “…turn to augment the NMT model with language-specific modules, e.g., the language-specific attention module (Blackwood et al., 2018), decoupled multilingual encoders and/or decoders (Vázquez et al., 2019; Escolano et al., 2020) and the lightweight language adapters (Bapna and Firat, 2019).”
Gender Bias in Multilingual Neural Machine Translation: The Architecture Matters (2020, background): “Language-Specific encoders-decoders (Escolano et al., 2020) exhibit less bias than the Shared encoder-decoder (Johnson et al.”
Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (2021, methodology): “…2019), broadening encoder outputs and normalizations (Zhang et al., 2020b), decoupling multilingual encoders and/or decoders (Vázquez et al., 2019; Escolano et al., 2020), using a fixed mix of LS and shared parameters (Wang et al., 2018), inserting lightweight adapters (Bapna & Firat, 2019) and…”
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (2020, background): “…effect of data (Arivazhagan et al., 2019; Wang et al., 2020a), architecture (Blackwood et al., 2018; Sachan & Neubig, 2018; V ´ azquez et al., 2019; Escolano et al., 2020) and scale (Huang et al., 2019b; Lepikhin et al., 2020) on multilingual models, their optimization dynamics are not well…”