Investigating Multilingual NMT Representations at Scale

By Sneha Kudugunta, Ankur Bapna, Isaac Caswell, Naveen Arivazhagan, Orhan Firat. 2019

138 citations recorded by the source corpus.

Multilingual Neural Machine Translation (NMT) models have yielded large empirical success in transfer learning settings. However, these black-box representations are poorly understood, and their mode of transfer remains elusive. In this work, we attempt to understand massively multilingual NMT representations (with 103 languages) using Singular Value Canonical Correlation Analysis (SVCCA), a representation similarity framework that allows us to compare representations across different languages, layers and models. Our analysis validates several empirical results and long-standing intuitions, and unveils new observations regarding how representations evolve in a multilingual translation model. We draw three major results from our analysis, with implications on cross-lingual transfer learning: (i) Encoder representations of different languages cluster based on linguistic similarity, (ii) Representations of a source language learned by the encoder are dependent on the target language, and vice-versa, and (iii) Representations of high resource and/or linguistically similar languages are more robust when fine-tuning on an arbitrary language pair, which is critical to determining how much cross-lingual transfer can be expected in a zero or few-shot setting. We further connect our findings with existing empirical observations in multilingual NMT and transfer learning.

Subfields: Artificial Intelligence, General Social Sciences, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Natural Language Processing

Citations in the corpus

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, background): “A multilingual version of EXMIX could provide a more robust foundation for the analysis of task families in existing works that analyze how multilingual NLP models transfer amongst different languages (Kudugunta et al., 2019; Hu et al., 2020; Wang et al., 2021).”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, background): “By training a shared set of (usually attention-based) parameters on many languages, these models can learn crosslingually aligned representations of text or speech in a shared representation space (Kudugunta et al., 2019; Wu et al., 2019).”
MURAL: Multimodal, Multitask Retrieval Across Languages (2021, background): “influenced from areal linguistics and contact linguistics, in addition to previously shown genealogical relationships (Kudugunta et al., 2019).”
Emerging Cross-lingual Structure in Pretrained Language Models (2019, methodology): “Kudugunta et al. (2019) use SVCCA to investigate the multilingual representations obtained by the encoder of a massively multilingual neural machine translation system (Aha-roni et al., 2019).”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, background): “Despite evidence that intermediate representations are not truly language-agnostic (Kudugunta et al., 2019), zero-shot translation has been shown successful, especially between related languages (Johnson et al.”
Are Multilingual Neural Machine Translation Models Better at Capturing Linguistic Features? (2020, background): “Kudugunta et al. (2019) investigate massively multilingual NMT on a combination of 103 languages.”
Finding Universal Grammatical Relations in Multilingual BERT (2020, methodology): “Both Libovick`y et al. (2019) and Kudugunta et al. (2019) perform SVCCA on LM representations extracted from mBERT and a massively multilingual transformer-based NMT model, ﬁnding language family-like clusters.”
Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth (2020, background): “…& Lampinen, 2020), and dynamics (Maheswaranathan et al., 2019) to revealing hidden linguistic structure in large language models (Bau et al., 2019; Kudugunta et al., 2019; Wu et al., 2019; 2020) to applications in neuroscience (Shi et al., 2019; Li et al., 2019; Merel et al., 2019; Zhang &…”
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (2020, background): “…the cross-lingual ability of these models and what contributes to it (Pires et al., 2019; Wu & Dredze, 2019; Wu et al., 2019; Artetxe et al., 2019; Kudugunta et al., 2019; Karthikeyan et al., 2020), the limitation of such training paradigm (Arivazhagan et al., 2019; Wang et al., 2020b), and how…”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “For multi-NMT, there have been experiments [91, 107] that delve into models to examine how the learning takes place and how diferent languages behave.”
How Reliable are Model Diagnostics? (2021, background): “, 2017; Hewitt and Manning, 2019; Marvin and Linzen, 2018), multilingual capabilities (Pires et al., 2019; Kudugunta et al., 2019), world knowledge (Jiang et al.”
Do Vision Transformers See Like Convolutional Neural Networks? (2021, unknown): “These techniques have been very successful in providing insights on properties of different vision architectures [29, 22, 18], representation structure in language models [48, 25, 47, 21], dynamics of training methods [33, 24] and domain specific model behavior [27, 35, 38].”
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021, background): “Multilingual models learn joint representations across languages to the extent of the parameters being shared (Wu and Dredze, 2019; Tiedemann, 2018; Tan et al., 2019; Zhang et al., 2020; Östling and Tiedemann, 2016; Kudugunta et al., 2019).”
Consistency is the key to further mitigating catastrophic forgetting in continual learning (2022, background): “The menace of catastrophic forgetting is not limited to CL alone, but manifests in multitask learning (Kudugunta et al., 2019), and supervised learning under domain shift (Ovadia et al., 2019) as well.”
A Survey of Deep Learning for Scientific Discovery (2020, methodology): “[118] In work on analyzing multilingual translation systems [118], representational analysis techniques are used to compute similarity of neural network (Transformer) hidden representations across diﬀerent languages.”
Transfusion: Understanding Transfer Learning for Medical Imaging (2019, methodology): “A recent tool that effectively overcomes these challenges is (Singular Vector) Canonical Correlation Analysis, (SV)CCA [22, 19], which has been used to study latent representations through training, across different models, alternate training objectives, and other properties [22, 19, 25, 18, 8, 17, ”
A Survey of Multilingual Neural Machine Translation (2019, methodology): “While the visualization of the embeddings of NMT models in multilingual settings has revealed some interesting behaviors [70, 83], we feel that more researchers should pursue this.”
Mixture of Languages: Improved Multilingual Encoders Through Language Grouping (2025, unknown): “Previous studies on MT have established that the first and last layers typically contain more language information (Kudugunta et al., 2019).”