Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation

By Chao Xing, Dong Wang, Chao Liu, Yiye Lin. 2015

431 citations recorded by the source corpus.

Word embedding has been found to be highly powerful to translate words from one language to another by a simple linear transform. However, we found some inconsistence among the objective functions of the embedding and the transform learning, as well as the distance measurement. This paper proposes a solution which normalizes the word vectors on a hypersphere and constrains the linear transform as an orthogonal transform. The experimental results confirmed that the proposed solution can offer better performance on a word similarity task and an English-toSpanish word translation task.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Cross-lingual Language Model Pretraining (2019, background): “A series of follow-up studies show that cross-lingual representations can be used to improve the quality of monolingual representations [16], that orthogonal transformations are sufficient to align these word distributions [43], and that all these techniques can be applied to an arbitrary number of ”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “For instance, Zhang et al. (2016) show that cross-lingual embeddings can be used to extend an English part-of-speech tagger to the cross-lingual setting, and Xiao and Guo (2014) achieve similar results in dependency parsing.”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, background): “A series of followup studies focus on aligning the word representation across languages (Xing et al., 2015; Ammar et al., 2016; Smith et al., 2017; Lample et al., 2018b).”
Word Translation Without Parallel Data (2017, background): “Xing et al. (2015) showed that these results are improved by enforcing an orthogonality constraint on W .”
Learning bilingual word embeddings with (almost) no bilingual data (2017, background): “Xing et al. (2015) incorporate length normalization in the training of word embeddings and maximize the cosine similarity instead, enforcing the orthogonality constraint to preserve the length normalization after the mapping.”
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks (2019, methodology): “Mikolov et al. (2013) has been applied small dictionaries to align word representations from different languages and it is sufﬁcient to align different languages with orthogonal transformation (Xing et al., 2015), even without parallel data (Lample et al., 2018).”
Offline bilingual word vectors, orthogonal transformations and the inverted softmax (2017, methodology): “It was recently independently proposed by Artetxe et al. (2016), and provides a numerically exact solution to the cost function proposed by Xing et al. (2015), just as the method of least squares provides a numerically exact solution to the cost function of Mikolov et al. (2013a).”
A Survey of Cross-lingual Word Embedding Models (2017, background): “This constraint is motivated by Xing et al. (2015) to preserve length normalization.”
Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces (2018, methodology): “Inspired by the notion, several studies have focused on designing algorithms that exploit this similarity to learn a cross-lingual alignment between the embedding spaces of two languages, where the two embedding spaces are trained from independent text corpora [6, 7, 8, 9, 10, 11, 12].”
A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings (2018, methodology): “…Lu et al., 2015), orthogonal methods, which map the embeddings in one or both languages under the constraint of the transformation being orthogonal (Xing et al., 2015; Artetxe et al., 2016; Zhang et al., 2016; Smith et al., 2017), and margin meth-ods, which map the embeddings in one language to…”
Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer (2020, background): “However, Xing et al. (2015) argue that there are some inconsistencies in directly learning the linear mapping.”
Explicit Alignment Objectives for Multilingual Bidirectional Encoders (2020, background): “…challenge dating back to the early stage of research in word alignment (Brown et al., 1993), cross-lingual embeddings (Faruqui and Dyer, 2014; Xing et al., 2015; Devlin et al., 2019; Conneau et al., 2020) are highly promising in their easy integration into neural network models for a variety…”
Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages (2019, methodology): “…n , which concatenate the pooled representations of all n sentences for each side in the source-pivot corpus, we rewrite Equation 1 as: which can be easily computed by the singular value decomposition (SVD) for a closed-form so-lution, if we put an orthogonality constraint on M (Xing et al., 2015).”
Cross-Lingual Entity Alignment via Joint Attribute-Preserving Embedding (2017, background): “(3) and the cosine similarity to measure embeddings [24].”
Dictionary-based Data Augmentation for Cross-Domain Neural Machine Translation (2020, methodology): “This study applies existing supervised (Xing et al., 2015) and unsupervised (Lample et al., 2018) lexicon induction methods to learn a lexicon.”
Ultradense Word Embeddings by Orthogonal Transformation (2016, methodology): “DENSIFIER is independent of the embedding learning algorithm and therefore extensible to other word embedding models like GloVe (Pennington et al., 2014), to phrase embeddings (Yu and Dredze, 2015) and even to sentence embeddings (Kiros et al., 2015).”
Earth Mover’s Distance Minimization for Unsupervised Bilingual Lexicon Induction (2017, methodology): “Later, proposals for using an orthogonal transformation are supported empirically (Xing et al., 2015; Zhang et al., 2016c; Artetxe et al., 2016) and theoretically (Smith et al., 2017).”