Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion

By Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Herv'e J'egou, Edouard Grave. Conference on Empirical Methods in Natural Language Processing, 2018

327 citations recorded by the source corpus.

Continuous word representations learned separately on distinct languages can be aligned so that their words become comparable in a common space. Existing works typically solve a quadratic problem to learn a orthogonal matrix aligning a bilingual lexicon, and use a retrieval criterion for inference. In this paper, we propose an unified formulation that directly optimizes a retrieval criterion in an end-to-end fashion. Our experiments on standard benchmarks show that our approach outperforms the state of the art on word translation, with the biggest improvements observed for distant language pairs such as English-Chinese.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/54b6d0daa0476919d209cbc4a045fe2a27f319fd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Zero-shot Cross-lingual Dialogue Systems with Transferable Latent Variables (2019, methodology): “Hence, as shown in Figure 1, we propose to reﬁne the cross-lingual word embeddings (Joulin et al., 2018) 1 using very few parallel word pairs, which is obtained by selecting 11 English words related to dialogue domains (weather, alarm, and reminder) and translate them using bilingual lexi-cons.”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “This alignment is typically done by computing a mapping between two different embedding spaces, such that the words in both embeddings share similar feature vectors after the mapping (Mikolov et al., 2013; Joulin et al., 2018).”
Bilingual Lexicon Induction through Unsupervised Machine Translation (2019, methodology): “…the hubness problem that arises when directly inducing bilingual dictionaries from cross-lingual embeddings, either through the retrieval method (Dinu et al., 2015; Smith et al., 2017; Conneau et al., 2018) or the mapping itself (Lazaridou et al., 2015; Shigeto et al., 2015; Joulin et al., 2018).”
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “However, to our knowledge, none of existing work has conducted large-scale evaluations of ChatGPT for multiple and diverse languages/tasks as we do.”
Identifying Fluently Inadequate Output in Neural and Statistical Machine Translation (2019, methodology): “We use the pre-trained aligned Wikipedia fastText word vectors (Joulin et al., 2018; Bojanowski et al., 2017).”
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale (2020, background): “News Feed and Search ranking [22], [50], are the most frequentlytrained models while language translation uses recurrent neural network (RNN) variants [14], [28], [45], and image classification and object detection/tracking use convolutional neural network (CNN) variants [23], [27], [35], [53].”
Density Matching for Bilingual Word Embedding (2019, methodology): “Early work focused on supervised methods maximizes the similarity of the embeddings of words that exist in a manually-created dictionary, according to some similarity metric (Mikolov et al., 2013; Faruqui and Dyer, 2014; Jawanpuria et al., 2018; Joulin et al., 2018).”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “Joulin et al. (2018) propose to optimize this metric directly when learning the mapping, as noted above.”
A Study of Neural Matching Models for Cross-lingual IR (2020, methodology): “The increasing popularity of projection-based weakly-supervised [4, 6, 14] and unsupervised [1, 2] cross-lingual word embeddings has spurred unsupervised frameworks [8] for CLIR, while in the realm of monolingual IR, interaction-based neural matching models [5, 10, 15] that utilize semantics contain”
Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer (2020, methodology): “the publicly available fastText embeddings trained on 294 languages in Wikipedia (Bojanowski et al., 2017; Joulin et al., 2018).”
Cross-Lingual Training of Neural Models for Document Ranking (2020, methodology): “For the token-based translation method, we used the MUSE bilingual dictionary (Lample et al., 2018) and the aligned word embeddings from fast-Text (Joulin et al., 2018).”
Word Embeddings and Gender Stereotypes in Swedish and English (2019, background): “Using aligned word vectors [21] would most likely help.”
A Survey of Model Architectures in Information Retrieval (2025, unknown): “Interaction-based models with cross-lingual word embeddings (Joulin et al., 2018) for cross-lingual reranking have also been explored (Yu and Allan, 2020).”