Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Models

By Chris Hokamp, John Glover, Demian Gholipour Ghalandari. 2019

15 citations recorded by the source corpus.

We study several methods for full or partial sharing of the decoder parameters of multi-lingual NMT models. Using only the WMT 2019 shared task parallel datasets for training, we evaluate both fully supervised and zero-shot translation performance in 110 unique translation directions. We use additional test sets and re-purpose evaluation methods recently used for unsupervised MT in order to evaluate zero-shot translation performance for language pairs where no gold-standard parallel data is available. To our knowledge, this is the largest evaluation of multi-lingual translation yet conducted in terms of the total size of the training data we use, and in terms of the number of zero-shot translation pairs we evaluate. We conduct an in-depth evaluation of the translation performance of different models, highlighting the trade-offs between methods of sharing decoder parameters. We find that models which have task-specific decoder parameters outperform models where decoder parameters are fully shared across all tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…pairs due to joint training and consequent positive transfer from higher-resource languages (Zoph et al., 2016; Firat et al., 2016b; Nguyen and Chiang, 2017; Johnson et al., 2017; Neubig and Hu, 2018; Aharoni et al., 2019; Escolano et al., 2019; Arivazhagan et al., 2019; Hokamp et al., 2019).”
Findings of the 2019 Conference on Machine Translation (WMT19) (2019, methodology): “2.5.4 AYLIEN_MULTILINGUAL (Hokamp et al., 2019) The Aylien research team built a Multilingual NMT system which is trained on all WMT2019 language pairs in all directions, then fine-tuned for a small number of iterations on Gujarati-English data only, including some self-backtranslated data.”
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021, background): “Strategies involve allocating language specific hidden states, attention modules, decoders or additional specialized layers ((Hokamp et al., 2019; Wang et al., 2018; Gu et al., 2018; Bapna et al., 2019)).”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “[83] showed that a unique decoder for each target language or unique decoder attention parameters for each target language outperforms models with fully shared decoder parameters.”
A Survey of Multilingual Neural Machine Translation (2019, background): “[64] showed that in a shared decoder setting, using a taskspecific (language pair to be translated) embedding works better than using language tokens.”