Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond

By Mikel Artetxe, Holger Schwenk. 2018

1,159 citations recorded by the source corpus.

Abstract We introduce an architecture to learn joint multilingual sentence representations for 93 languages, belonging to more than 30 different families and written in 28 different scripts. Our system uses a single BiLSTM encoder with a shared byte-pair encoding vocabulary for all languages, which is coupled with an auxiliary decoder and trained on publicly available parallel corpora. This enables us to learn a classifier on top of the resulting embeddings using English annotated data only, and transfer it to any of the 93 languages without any modification. Our experiments in cross-lingual natural language inference (XNLI data set), cross-lingual document classification (MLDoc data set), and parallel corpus mining (BUCC data set) show the effectiveness of our approach. We also introduce a new test set of aligned sentences in 112 languages, and show that our sentence embeddings obtain strong results in multilingual similarity search even for low- resource languages. Our implementation, the pre-trained encoder, and the multilingual test set are available at https://github.com/facebookresearch/LASER.

Subfields: Artificial Intelligence, General Health Professions, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Health Professions

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, background): “In another relevant work, Artetxe and Schwenk (2018) trained an NMT model on 93 languages and used the learned representations to perform cross-lingual transfer learning.”
Cross-lingual Language Model Pretraining (2019, methodology): “We report the XNLI baselines of Conneau et al. (2018b), the multilingual BERT approach of De-vlin et al. (2018) and the recent work of Artetxe and Schwenk (2018).”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, background): “, 2018b, 2019), error correction (Chollampatt and Ng, 2018), multilingual sentence embeddings (Artetxe and Schwenk, 2018), and dialogue (Miller et al.”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “We focus on the embeddings generated by the LASER encoder, which enables the comparison of sentences in 94 different languages (Artetxe and Schwenk, 2018a).”
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, methodology): “2018), or by limiting the decoder to use a pooled vector representation of the entire sentence from the encoder (Artetxe and Schwenk, 2018).”
Neural Machine Translation for English–Kazakh with Morphological Segmentation and Synthetic Data (2019, methodology): “LMVR was trained on the KK side of the RU–KK parallel data as well as on the monolingual KK data.”
MLQA: Evaluating Cross-lingual Extractive Question Answering (2019, background): “The reader is referred to Artetxe and Schwenk (2018) and Artetxe and Schwenk (2019) for a detailed description.”
Self-training Improves Pre-training for Natural Language Understanding (2020, methodology): “We use a large bank of web documents and construct sentence embeddings (Kiros et al., 2015; Wieting et al., 2016; Conneau et al., 2017; Artetxe and Schwenk, 2019; Cer et al., 2018; Arora et al., 2017) that allow us to retrieve domain-speciﬁc unannotated sentences, which are similar to the existing…”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, methodology): “[2] found that using two target languages was enough to make the sentence embedding space aligned.”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, methodology): “Some of these difﬁculties may be due to to-kenisation and an under-representation of ideograms in the joint sentencepiece vocabulary, which has been shown to be important in a cross-lingual model’s performance (Artetxe et al., 2019; Conneau et al., 2019).”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, positive): “with cross-lingual LASER embeddings (Artetxe and Schwenk, 2019).”
How Multilingual is Multilingual BERT? (2019, methodology): “will likely require the model to incorporate an explicit multilingual training objective, such as that used by Lample and Conneau (2019) or Artetxe and Schwenk (2018).”
The State and Fate of Linguistic Diversity and Inclusion in the NLP World (2020, background): “Neural systems, on one hand, require a lot more data for training than rule-based or traditional ML systems, creating a bigger technological divide between the Xs and Ys; yet, some of the most recent techniques on zero-shot learning of massively multilingual systems (Devlin et al., 2019; Conneau and”
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks (2019, methodology): “Artetxe and Schwenk (2018) use multilingual machine translation to train a multilingual sentence encoder and use this fixed sentence embedding to classify XNLI.”
Pre-training via Paraphrasing (2020, methodology): “Better results have been achieved on both tasks using labeled bitext for training [Artetxe and Schwenk, 2019], but our results suggest that our pre-training objective learns an effective cross-lingual retrieval function.”
FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP (2019, background): “, 2017) and LASER embeddings (Artetxe and Schwenk, 2018), and expanding our coverage of NLP datasets and formats for automatic data fetching.”
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, methodology): “We conduct extensive experiments on several cross-lingual understanding tasks, including cross-lingual natural language inference (Con-neau et al., 2018), cross-lingual question answering (Lewis et al., 2020), and cross-lingual sentence retrieval (Artetxe and Schwenk, 2019).”
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (2019, background): “Concurrent to our work, Artetxe and Schwenk (2018) use bitext between English/Spanish and the rest of languages to pretrain a multilingual sentence representation with a sequence-to-sequence model where the decoder only has access to a max-pooling of the encoder hidden states. mBERT outperforms…”