Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation

By Akiko Eriguchi, Melvin Johnson, Orhan Firat, Hideto Kazawa, Wolfgang Macherey. 2018

65 citations recorded by the source corpus.

Transferring representations from large supervised tasks to downstream tasks has shown promising results in AI fields such as Computer Vision and Natural Language Processing (NLP). In parallel, the recent progress in Machine Translation (MT) has enabled one to train multilingual Neural MT (NMT) systems that can translate between multiple languages and are also capable of performing zero-shot translation. However, little attention has been paid to leveraging representations learned by a multilingual NMT system to enable zero-shot multilinguality in other NLP tasks. In this paper, we demonstrate a simple framework, a multilingual Encoder-Classifier, for cross-lingual transfer learning by reusing the encoder from a multilingual NMT system and stitching it with a task-specific classifier component. Our proposed model achieves significant improvements in the English setup on three benchmark tasks - Amazon Reviews, SST and SNLI. Further, our system can perform classification in a new language for which no classification data was seen during training, showing that zero-shot classification is possible and remarkably competitive. In order to understand the underlying factors contributing to this finding, we conducted a series of analyses on the effect of the shared vocabulary, the training data type for NMT, classifier complexity, encoder representation power, and model generalization on zero-shot performance. Our results provide strong evidence that the representations learned from multilingual NMT systems are widely applicable across languages and tasks.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, background): “…for future work, including semi-supervised learning in such settings, exploring ways to reduce the performance degradation when increasing the number of languages, or using such models for multilingual transfer learning (McCann et al., 2017; Eriguchi et al., 2018; Artetxe and Schwenk, 2018).”
Cross-lingual Language Model Pretraining (2019, methodology): “By using parallel data, Hermann and Blunsom (2014); Conneau et al. (2018b); Eriguchi et al. (2018) investigated zero-shot cross-lingual sentence classi-ﬁcation.”
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, background): “Our work is more closely related to two very recent works which explore the encoder from multilingual NMT model for cross-lingual transfer learning (Eriguchi et al., 2018; Artetxe and Schwenk, 2018).”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, background): “Past work also leverages supervised ASR data to build multilingual representations of speech (Kannan et al., 2019; Bai et al., 2021), similar to how multilingual machine translation in NLP is used to build multilingual representations (Eriguchi et al., 2018; Siddhant et al., 2020).”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “…2018; Peters et al., 2018; Devlin et al., 2019), multilingual extensions of pretrained encoders have recently been shown to be effective for learning deep cross-lingual representations (Eriguchi et al., 2018; Pires et al., 2019; Wu & Dredze, 2019; Lample & Conneau, 2019; Siddhant et al., 2019).”
Investigating Multilingual NMT Representations at Scale (2019, background): “…on tasks including lowresource language translation (Zoph et al., 2016; Nguyen and Chiang, 2017; Neubig and Hu, 2018) and zero or few-shot transfer learning for downstream tasks (Eriguchi et al., 2018; Lample and ∗Work done as part of Google AI Residency Conneau, 2019; Wu and Dredze, 2019).”
Aligning Cross-Lingual Entities with Multi-Aspect Information (2019, background): “Recent works (Hermann and Blunsom, 2014; Conneau et al., 2018; Eriguchi et al., 2018) studied crosslingual sentence classification via zero-shot learning.”
(Self-Attentive) Autoencoder-based Universal Language Representation for Machine Translation (2018, methodology): “On the other hand, our work is also related to recent works on sentence representations (Conneau et al., 2017, 2018; Eriguchi et al., 2018).”
Explicit Alignment Objectives for Multilingual Bidirectional Encoders (2020, methodology): “On the other hand, some methods directly leverage multilingual parallel corpora (McCann et al., 2017; Eriguchi et al., 2018; Conneau and Lample, 2019; Huang et al., 2019; Siddhant et al., 2020), which gives some degree of supervision implicitly aligning the words in the two languages.”
Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference (2020, background): “Moreover, recent work is investigating cross-lingual NLI (Eriguchi et al., 2018; Conneau et al., 2018), and this is encouraging to consider multilinguality in future work.”
Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog (2018, methodology): “…also a lot of work of using crosslingual sequence encoders for sentence classifications using either multilingual MT encoders similar to ours (e.g., Eriguchi et al., 2018; Yu et al., 2018a; Singla et al., 2018) or training encoders and then aligning their vector spaces after pretraining (Conneau…”
Towards Interlingua Neural Machine Translation (2019, methodology): “On the other hand, our work is also related to recent works on sentence representations (Conneau et al. 2017, 2018; Eriguchi et al. 2018).”
MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages (2022, background): “More recently, researchers have borrowed encoders from pre-trained neural translation models before building subsequent classifiers and NER models (Eriguchi et al., 2018; Schuster et al., 2019), also focusing on language-agnostic and language specific features to learn what information to share…”