Exploiting Similarities among Languages for Machine Translation

By Tomas Mikolov, Quoc V. Le, Ilya Sutskever. 2013

1,653 citations recorded by the source corpus.

Dictionaries and phrase tables are the basis of modern statistical machine translation systems. This paper develops a method that can automate the process of generating and extending dictionaries and phrase tables. Our method can translate missing word and phrase entries by learning language structures based on large monolingual data and mapping between languages from small bilingual data. It uses distributed representation of words and learns a linear mapping between vector spaces of languages. Despite its simplicity, our method is surprisingly effective: we can achieve almost 90% precision@5 for translation of words between English and Spanish. This method makes little assumption about the languages, so it can be used to extend and refine dictionaries and translation tables for any language pairs.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Skip-Thought Vectors (2015, methodology): “Inspired by [15], which learned linear mappings between translation word spaces, we solve an un-regularized L2 linear regression loss for the matrix W .”
Distributed Representations of Sentences and Documents (2014, methodology): “Our technique is inspired by the recent work in learning vector representations of words using neural networks (Bengio et al., 2006; Collobert & Weston, 2008; Mnih & Hinton, 2008; Turian et al., 2010; Mikolov et al., 2013a;c).”
Multilingual Denoising Pre-training for Neural Machine Translation (2020, background): “for various translation applications. Multilinguality in NLP tasks This work is also related to the continual trend of multilingual language learning, including aligning multilingual word embeddings (Mikolov et al., 2013; Chen and Cardie, 2018; Lample et al., 2018b) into universal space, and learnin”
Cross-lingual Language Model Pretraining (2019, methodology): “Aligning distributions of text representations has a long tradition, starting from word embed-dings alignment and the work of Mikolov et al. (2013a) that leverages small dictionaries to align word representations from different languages.”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, methodology): “More sophisticated approaches—such as the unsupervised SkipThought model of Kiros et al. (2015) that extends the skip-gram model of Mikolov et al. (2013b) to the sentence level—have been proposed to capture syntactic and semantic dependencies inside sentence representations.”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, background): “Mikolov et al. (2013a) ﬁrst introduces dictionaries to align word representations from different languages.”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, background): “Cross-Lingual Representations Various previous works have explored leveraging cross-lingual word representations to build parallel dictionaries and phrase tables, then applying them to downstream tasks [1, 2, 3, 4, 24, 29].”
Unsupervised Speech Recognition (2021, background): “Mikolov et al. (2013a) aligned word2vec embedding spaces (Mikolov et al., 2013b) by building a linear mapping from a source language to a target language using supervision from a seed dictionary of word translation pairs.”
Unsupervised Cross-lingual Representation Learning at Scale (2019, background): “Parallel work on cross-lingual understanding (Mikolov et al., 2013a; Schuster et al., 2019; Lample and Conneau, 2019) extends these systems to more languages and to the cross-lingual setting in which a model is learned in one language and applied in other languages.”
Recurrent Neural Network Regularization (2014, methodology): “…recognition (Robinson et al., 1996; Graves et al., 2013) and have recently been applied to machine translation, where they are used for language modeling, re-ranking, or phrase modeling (Devlin et al., 2014; Kalchbrenner & Blunsom, 2013; Cho et al., 2014; Chow et al., 1987; Mikolov et al., 2013).”
Unsupervised Neural Machine Translation (2017, methodology): “Given that the proposed system uses pre-trained cross-lingual embeddings in the encoder as described in Section 3.1, we use the monolingual corpora described above to independently train the embeddings for each language using word2vec (Mikolov et al., 2013b).”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “Over the last few years, there has been a move towards general-purpose multilingual representations that are applicable to many tasks, both on the word level (Mikolov et al., 2013; Faruqui & Dyer, 2014; Artetxe et al., 2017) or the full-sentence level (Devlin et al.”
Word Translation Without Parallel Data (2017, methodology): “Most successful approaches for learning cross-lingual word embeddings have used bilingual dictionaries to train the mapping from the source to the target space Mikolov et al. (2013b); Zou et al. (2013); Faruqui & Dyer (2014); Ammar et al. (2016).”
Text Understanding from Scratch (2015, methodology): “One technique which draws great interests is word2vec(Mikolov et al., 2013b).”
Deep Learning: Methods and Applications (2014, background): “Yet another deep learning approach to machine translation appeared in [249].”
Learning bilingual word embeddings with (almost) no bilingual data (2017, methodology): “…the embeddings for each language on monolingual corpora, and then learn a linear transformation to map the embeddings from one space into the other by minimizing the distances in a bilingual dictionary, usually in the range of a few thousand entries (Mikolov et al., 2013a; Artetxe et al., 2016).”
Taskonomy: Disentangling Task Transfer Learning (2018, background): “s performing the learning at a level higher than where conventional learning occurs, e.g. as employed in reinforcement learning [21,31,28], optimization [2,82,48], or certain architectural mechanisms [27,30,87,65]. The motivation behind meta learning has similarities to ours and our outcome can be s”
Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors (2014, methodology): “Top accuracy on the entire data set (an) and on the semantic subset (ansem) was reached by Mikolov et al. (2013c) using a skip-gram predict model.”
Learning Continuous Phrase Representations for Translation Modeling (2014, methodology): “Continuous space models have also been used for generating translations for new words (Mikolov et al. 2013a) and ITG reordering (Li et al. 2013).”