When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation?

By Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Padmanabhan, Graham Neubig. 2018

371 citations recorded by the source corpus.

The performance of Neural Machine Translation (NMT) systems often suffers in low-resource scenarios where sufficiently large-scale parallel corpora cannot be obtained. Pre-trained word embeddings have proven to be invaluable for improving performance in natural language analysis tasks, which often suffer from paucity of data. However, their utility for NMT has not been extensively explored. In this work, we perform five sets of experiments that analyze when we can expect pre-trained word embeddings to help in NMT tasks. We show that such embeddings can be surprisingly effective in some cases – providing gains of up to 20 BLEU points in the most favorable setting.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “Qi et al. (2018) investigated the application of pre-trained word embeddings for MT; Ramachandran et al. (2017) proposed to pre-train the encoder-decoder modules as two separate language models.”
Massively Multilingual Neural Machine Translation (2019, methodology): “Our experiments on the publicly available TED talks dataset (Qi et al., 2018) show that massively multilingual many-to-many models with up to 58 languages to-and-from English are very effective in low resource settings, allowing to use high-capacity models while avoiding overfitting and achieving…”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “The most commonly used datasets include: ( i ) TED talks (Cettolo et al., 2012) which includes 59 languages (Qi et al., 2018) with around 3k to 200k sentence pairs per Figure 1: Per language pair data distribution of the training dataset used for our multilingual experiments.”
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, methodology): “…weights from an auxiliary task is a widely used strategy that has shown to improve the performance in tasks such as text classiﬁcation (Dai and Le 2015; Howard and Ruder 2018), question-answering (Devlin et al. 2018), and machine translation (Ramachandran, Liu, and Le 2017; Qi et al. 2018).”
Beyond English-Centric Multilingual Machine Translation (2020, background): “• TED — The TED Talks dataset3 (Ye et al., 2018) contains translations between more than 50 languages; most of the pairs do not include English.”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, neutral): “, 2013) and TED59 (Qi et al., 2018) for ar, fa, id, it, nl, pt, sl, sv, leaving ca and cy unpaired.”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, methodology): “The quality of the extracted parallel sentences was evaluated in a machine translation (MT) task on six European (da, cr, sl, sk, lt, et) languages of the TED corpus (Qi et al., 2018), where it compared favorably to systems built on crawled sentences from WikiMatrix and ParaCrawl v6.”
Nearest Neighbor Machine Translation (2020, methodology): “T ED T ALKS : We use the Ted Talks data prepared by Qi et al. (2018) for evaluation in the multilingual setting, particularly to explore performance for language pairs that do no include English.”
Masked Language Model Scoring (2019, methodology): “We consider 5 low-resource directions from the TED Talks dataset (Qi et al., 2018): Arabic (ar), Galician (gl), and Slovak (sk) to English; and English to Arabic, German (de), languages which were considered in Aharoni et al. (2019).”
Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution (2021, background): “Neu-big and Hu (2018) showed that a large multilingual translation model trained on a subset of languages of the TED dataset (Qi et al., 2018) could perform translation on the remaining (related) languages.”
Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020, background): “Qi et al. (2018) analyzes when the pre-trained word embeddings can help in NMT tasks.”
AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “We would like to thank Nobuyuki Morioka and Yifan Ding for their help in re-creating the TTSaugmented WMT/TED dataset which was also used in Jia et al. [2022a] and Adam Roberts and Ron Weiss for their advice and reviews.”
Multilingual Neural Machine Translation with Knowledge Distillation (2019, methodology): “Ted Talk: We use the common corpus of TED talk which contains translations between multiple languages (Ye et al., 2018).”
Transformers without Tears: Improving the Normalization of Self-Attention (2019, methodology): “Talks (Qi et al., 2018) and IWSLT '15 (Cettolo et al., 2015) corpora, we ﬁrst train state-of-the-art Transformer models (+4.0 BLEU on average over the best published NMT bitext-only numbers).”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “(Aharoni et al., 2019; Arivazhagan et al., 2019) train models to translate between English and 102 languages, for the 10 most high-resourced African languages on private data, and otherwise on public TED talks (Qi et al., 2018).”
Rapid Adaptation of Neural Machine Translation to New Languages (2018, methodology): “Like Qi et al. (2018), we experiment with Azerbaijani (aze), Belarusian (bel), and Galician (glg) to English, and also additionally add Slovak (slk), a slightly higher resourced language, for contrast.”
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia (2019, methodology): “We identified a publicly available data set which provide test sets for many language pairs: translations of TED talks as proposed in the context of a study on pretrained word embeddings for NMT14 (Qi et al., 2018).”