A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
By Mikel Artetxe, Gorka Labaka, Eneko Agirre. 2018
628 citations recorded by the source corpus.
Recent work has managed to learn cross-lingual word embeddings without parallel data by mapping monolingual embeddings to a shared space through adversarial training. However, their evaluation has focused on favorable conditions, using comparable corpora or closely-related languages, and we show that they often fail in more realistic scenarios. This work proposes an alternative approach based on a fully unsupervised initialization that explicitly exploits the structural similarity of the embeddings, and a robust self-learning algorithm that iteratively improves this solution. Our method succeeds in all tested scenarios and obtains the best published results in standard datasets, even surpassing previous supervised systems. Our implementation is released as an open source project at https://github.com/artetxem/vecmap.
Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Sebastian Ruder
- Kyunghyun Cho
- Percy Liang
- Yoshua Bengio
In Natural Language Processing
- Graham Neubig
- Sebastian Ruder
- Ivan Vulic
- Mohit Bansal
- Richard Socher
- Kyunghyun Cho
- Luke S. Zettlemoyer
- Hinrich Schütze
In Representation & Self-Supervised Learning
- Ivan Vulic
- Sebastian Ruder
- Graham Neubig
- Mohammad Taher Pilehvar
- Goran Glavas
- José Camacho-Collados
- Anders Søgaard
- Mikel Artetxe
In Machine Translation
- Ivan Vulic
- Sebastian Ruder
- Graham Neubig
- Mikel Artetxe
- Goran Glavas
- Eneko Agirre
- Gorka Labaka
- Anders Søgaard
Citations in the corpus
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “, 2018a), heuristic initialisation (Artetxe et al., 2018), and optimal transport (Zhang et al.”
- Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond (2018, neutral): “…that is becoming increasingly popular is to separately train word embeddings for each language, and map them to a shared space based on a bilingual dictionary (Mikolov et al., 2013a; Artetxe et al., 2018a) or even in a fully unsupervised manner (Conneau et al., 2018a; Artetxe et al., 2018b).”
- On the Cross-lingual Transferability of Monolingual Representations (2019, methodology): “We also include a method based on this alternative approach where we train skip-gram embeddings for each language, and map them to a shared space using VecMap (Artetxe et al., 2018).”
- Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (2019, background): “(2017) and Artetxe et al. (2018) first show two monolingual embeddings can be aligned by learning an orthogonal mapping with only identical strings as an initial heuristic bilingual dictionary.”
- Unsupervised Statistical Machine Translation (2018, methodology): “For instance, existing unsupervised NMT systems obtain between 15-16 BLEU points in WMT 2014 English-French translation, whereas a state-of-the-art NMT system obtains around 41 (Artetxe et al., 2018c; Lample et al., 2018; Yang et al., 2018).”
- A Survey of Cross-lingual Word Embedding Models (2017, methodology): “…Luan, & Sun, 2017b; Conneau et al., 2018a) and offer support to neural machine translation and cross-lingual information retrieval from monolingual data only (Lample, Denoyer, & Ranzato, 2018; Artetxe, Labaka, Agirre, & Cho, 2018; Litschko, Glavaš, Ponzetto, & Vulić, 2018; Artetxe et al., 2018a).”
- JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages (2019, methodology): “Based on the findings from Glavaˇs et al. (2019), we compare the most effective and the most robust unsupervised method of Artetxe et al. (2018) to a simple supervised method (Smith et al., 2017) in the bilingual lexicon induction task (BLI).”
- Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning (2020, background): “…a great many languages, cross-lingual language models (Devlin et al., 2019; Conneau and Lample, 2019; Huang et al., 2019) have significantly improved the cross-lingual performance upon the cross-lingual word embeddings (Conneau et al., 2017; Artetxe et al., 2018) based models (Wu and Dredze, 2019).”
- Unsupervised Cross-Lingual Representation Learning (2019, methodology): “We will put focus on a standardized two-step mappingbased framework (Artetxe et al., 2018a) that generalizes all mapping-based approaches, and analyze the importance of each component of the framework.”
- Visual Grounding in Video for Unsupervised Word Translation (2020, methodology): “Earlier work used various degrees of supervision through ground-truth dictionaries or heuristics [4, 26, 42]; recently, fully unsupervised approaches achieved a similar performance on word alignment for different language pairs without any supervision [6, 10].”
- SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation (2021, methodology): “We first concate-nate the monolingual corpora of two languages and learn joint BPE, and then train cross-lingual BPE embeddings with VecMap (Artetxe et al., 2018).”
- Probing Pretrained Language Models for Lexical Semantics (2020, methodology): “We learn “BERT-based” CLWEs using a standard mappingbased approach (Mikolov et al., 2013a; Smith et al., 2017) with VECMAP (Artetxe et al., 2018).”
- Massively Multilingual Transfer for NER (2019, background): “Recent advances in (un)supervised bilingual dictionary induction (Gouws and Søgaard, 2015; Duong et al., 2016; Lample et al., 2018; Artetxe et al., 2018; Schuster et al., 2019) have enabled cross-lingual alignment with bilingual dictionaries (Mayhew et al., 2017; Xie et al., 2018).”
- Should All Cross-Lingual Embeddings Speak English? (2019, methodology): “…is often the first step towards several down-stream tasks such as Part-Of-Speech (POS) tagging (Zhang et al., 2016), parsing (Ammar et al., 2016a), document classification (Klementiev et al., 2012), and machine translation (Irvine and Callison-Burch, 2013; Artetxe et al., 2018b; Lample et al., 2018).”
- Are We Consistently Biased? Multidimensional Analysis of Biases in Distributional Word Vectors (2019, background): “…of models for inducing cross-lingual embedding spaces (Mikolov et al., 2013a; Hermann and Blunsom, 2014; Smith et al., 2017; Conneau et al., 2018; Artetxe et al., 2018; Hoshen and Wolf, 2018, inter alia) and their ubiquitous application in cross-lingual transfer of NLP models for downstream…”
- Unsupervised Cross-lingual Word Embedding by Multilingual Neural Language Models (2018, methodology): “3: Some examples when Spanish and English words matched correctly by our model using 50k sentences, but not by Artetxe, Labaka, and Agirre (2018).”
- Finding and Characterizing Information Warfare Campaigns (2020, background): “Multilingual embeddings have been accomplished by learning jointly on parallel corpora [200] or by training independently and then mapping to a shared space with a bilingual dictionary [18].”
- SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings (2020, methodology): “Subsequently we use VecMap (Artetxe et al., 2018b) to map the embeddings into a common multilingual space.”