Improving Vector Space Word Representations Using Multilingual Correlation

By Manaal Faruqui, Chris Dyer. 2014

613 citations recorded by the source corpus.

The distributional hypothesis of Harris (1954), according to which the meaning of words is evidenced by the contexts they occur in, has motivated several effective techniques for obtaining vector space semantic representations of words using unannotated text corpora. This paper argues that lexico-semantic content should additionally be invariant across languages and proposes a simple technique based on canonical correlation analysis (CCA) for incorporating multilingual evidence into vectors generated monolingually. We evaluate the resulting word representations on standard lexical semantic evaluation tasks and show that our method produces substantially better semantic representations than monolingual techniques.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Cross-lingual Language Model Pretraining (2019, background): “A series of follow-up studies show that cross-lingual representations can be used to improve the quality of monolingual representations [16], that orthogonal transformations are sufficient to align these word distributions [43], and that all these techniques can be applied to an arbitrary number of ”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “Many of these methods require some form of supervision (typically in the form of a small bilingual lexicon) to align two sets of source and target embeddings to the same space (Mikolov et al., 2013a; Kociský et al., 2014; Faruqui and Dyer, 2014; Ammar et al., 2016).”
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, background): “Several recent efforts have explored methods that align vector spaces for words in different languages (Faruqui and Dyer, 2014; Upadhyay et al., 2016; Ruder et al., 2017).”
Towards Universal Paraphrastic Sentence Embeddings (2015, methodology): “We tune over the strategy in our experiments.”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “Over the last few years, there has been a move towards general-purpose multilingual representations that are applicable to many tasks, both on the word level (Mikolov et al., 2013; Faruqui & Dyer, 2014; Artetxe et al., 2017) or the full-sentence level (Devlin et al., 2019; Lample & Conneau, 2019).”
Word Translation Without Parallel Data (2017, background): “Since then, several works aimed at improving these cross-lingual word embeddings (Faruqui & Dyer (2014); Xing et al. (2015); Baroni et al. (2015); Ammar et al. (2016); Artetxe et al. (2016); Smith et al. (2017)), but they all rely on bilingual word lexicons.”
Visualizing and Understanding Neural Models in NLP (2015, background): “Similarity is commonly visualized graphically, generally by projecting the embedding space into two dimensions and observing that similar words tend to be clustered together (e.g., Elman (1989), Ji and Eisenstein (2014), Faruqui and Dyer (2014)).”
Learning bilingual word embeddings with (almost) no bilingual data (2017, background): “Instead of learning a single linear transformation from the source language into the target language, Faruqui and Dyer (2014) use canonical correlation analysis to map both languages to a shared vector space.”
Offline bilingual word vectors, orthogonal transformations and the inverted softmax (2017, methodology): “They can also be higher quality than monolingual vectors (Faruqui & Dyer, 2014).”
BilBOWA: Fast Bilingual Distributed Representations without Word Alignments (2014, methodology): “It was extended by Faruqui et al. [9], who simultanteously projected source and target language embeddings into a joint space using canonical correlation analysis.”
Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation (2018, background): “Obtaining representations from word level alignments - bilingual dictionaries or automatically generated word alignments - is the most popular approach (Mikolov et al. 2013; Faruqui and Dyer 2014; Zou et al. 2013).”
Massively Multilingual Word Embeddings (2016, methodology): “Some previous work on bilingual and multilingual embeddings focuses on monolingual word similarity to evaluate embeddings (e.g., Faruqui and Dyer, 2014).”
Pre-trained models for natural language processing: A survey (2020, background): “tions shared across languages plays an important role in many cross-lingual NLP tasks. Cross-Lingual Language Understanding (XLU) Most of the early works focus on learning multilingual word embedding [45, 123, 158], which represents text from multiple languages in a single semantic space. However, t”
A Primer on Neural Network Models for Natural Language Processing (2015, background): “This is likely to produce vectors that are similar to the window-based approach, but reducing the somewhat undesired effect of the window-based approach in which antonyms (e.g. hot and cold, high and low) tend to receive similar vectors (Faruqui & Dyer, 2014).”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, background): “We can also have multi-view situations even when the views have the same data modalities (textual, visual, audio), such as in the case of parallel texts in different languages [236, 79], photos of the same person taken across different poses, illuminations and expressions [261, 195, 196, 170].”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “Faruqui and Dyer (2014) later applied CCA to project words from two languages into a shared embedding space.”
Sparse Overcomplete Word Vector Representations (2015, methodology): “Faruqui and Dyer (2014) learned vectors by ﬁrst performing SVD on text in different languages, then applying canonical correlation analysis on pairs of vectors for words that align in parallel corpora.”
Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation (2015, background): “More importantly, the embedding encodes syntactic and semantic content implicitly, so that relations among words can be simply computed as the distances among their embeddings, or word vectors.”
Visualizing and Understanding Neural Machine Translation (2017, background): “Though projecting word embedding space into two dimensions (Faruqui and Dyer, 2014) and the attention matrix (Bahdanau et al.”
Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces (2018, methodology): “Inspired by the notion, several studies have focused on designing algorithms that exploit this similarity to learn a cross-lingual alignment between the embedding spaces of two languages, where the two embedding spaces are trained from independent text corpora [6, 7, 8, 9, 10, 11, 12].”