Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT

By Alexandra Chronopoulou, Dario Stojanovski, Alexander M. Fraser. 2020

36 citations recorded by the source corpus.

Using a language model (LM) pretrained on two languages with large monolingual data in order to initialize an unsupervised neural machine translation (UNMT) system yields state-of-the-art results. When limited data is available for one language, however, this method leads to poor translations. We present an effective approach that reuses an LM that is pretrained only on the high-resource language. The monolingual LM is fine-tuned on both languages and is then used to initialize a UNMT model. To reuse the pretrained LM, we have to modify its predefined vocabulary, to account for the new language. We therefore propose a novel vocabulary extension method. Our approach, RE-LM, outperforms a competitive cross-lingual pretraining model (XLM) in English-Macedonian (En-Mk) and English-Albanian (En-Sq), yielding more than +8.3 BLEU points for all four translation directions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, methodology): “, 2020) Exclusion Adapting for low-resource varieties (Chronopoulou et al., 2020; Kumar et al., 2021a)”
Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution (2021, background): “Past works have explored adapting translation models to new languages, typically focusing on related languages which share similar scripts (Gu et al., 2018; Neubig and Hu, 2018; Lakew et al., 2019; Chronopoulou et al., 2020).”
Machine Translation into Low-resource Language Varieties (2021, methodology): “Baselines On the unsupervised models based on Lample et al. (2018a), we observe a similar trend as that of Chronopoulou et al. (2020), where the L AMPLE -”
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022, background): “, 2021b) as well as NMT scenarios (Bapna and Firat, 2019; Philip et al., 2020; Chronopoulou et al., 2020; Le et al., 2021; Üstün et al., 2021; Stickland et al., 2021; Garcia et al., 2021).”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “[28] resolved this issue through TL by irst training the unsupervised NMT model on a high-resource language pair and further ine-tuning on the LRL pair.”