Bilingual Word Embeddings for Phrase-Based Machine Translation

By Will Y. Zou, Richard Socher, Daniel Matthew Cer, Christopher D. Manning. 2013

581 citations recorded by the source corpus.

We introduce bilingual word embeddings: semantic embeddings associated across two languages in the context of neural language models. We propose a method to learn bilingual embeddings from a large unlabeled corpus, while utilizing MT word alignments to constrain translational equivalence. The new embeddings significantly out-perform baselines in word semantic similarity. A single semantic similarity feature induced with bilingual embeddings adds near half a BLEU point to the results of NIST08 Chinese-English machine translation task.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, methodology): “One important difference between the proposed RNN Encoder–Decoder and the approaches in (Zou et al., 2013) and (Chandar et al.”
Distributed Representations of Sentences and Documents (2014, methodology): “, 2013), statistical machine translation (Mikolov et al., 2013b; Zou et al., 2013), image understanding (Frome et al.”
Learning Visual N-Grams from Web Data (2016, background): “Several prior studies have used phrase embeddings for natural language processing tasks such as named entity recognition [45], text classification [27, 53, 61], and machine translation [68, 71].”
Word Translation Without Parallel Data (2017, background): “ous cross-lingual language processing systems. More recently, several approaches have been proposed to learn bilingual dictionaries mapping from the source to the target space (Mikolov et al., 2013b; Zou et al., 2013; Faruqui 9 Published as a conference paper at ICLR 2018 & Dyer, 2014; Ammar et ”
Learning bilingual word embeddings with (almost) no bilingual data (2017, background): “In addition to having a direct application in inherently crosslingual tasks like machine translation (Zou et al., 2013) and crosslingual entity linking (Tsai and Roth, 2016), they provide an excellent mechanism for transfer learning, where a model trained in a resource-rich language is transferred…”
Improving Vector Space Word Representations Using Multilingual Correlation (2014, methodology): “Our method of learning multilingual word vectors is most closely associated to Zou et al. (2013) who learn bilingual word embeddings and show their utility in machine translation.”
BilBOWA: Fast Bilingual Distributed Representations without Word Alignments (2014, background): “…useful for transferring the limited label information from high-resource to low-resource languages, and has been demonstrated to be effective for document classification [12], outperforming a strong machine-translation baseline; as well as named-entity recognition and machine translation [21, 15].”
Multilingual Distributed Representations without Word Alignment (2013, background): “Using a slightly different approach, [29], also learned bilingual embeddings for machine translation.”
Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation (2018, background): “Obtaining representations from word level alignments - bilingual dictionaries or automatically generated word alignments - is the most popular approach (Mikolov et al. 2013; Faruqui and Dyer 2014; Zou et al. 2013).”
Massively Multilingual Word Embeddings (2016, background): “There is a rich body of literature on bilingual embeddings, including work on machine translation (Zou et al., 2013; Hermann and Blunsom, 2014; Cho et al., 2014; Luong et al., 2015b; Luong et al., 2015a, inter alia ), cross-lingual dependency parsing (Guo et al., 2015; Guo et al., 2016), and…”
Compositional Morphology for Word Representations and Language Modelling (2014, background): “Recent work has moved beyond monolingual vector-space modelling, incorporating phrase similarity ratings based on bilingual word embeddings as a translation model feature (Zou et al., 2013), or formulating translation purely in terms of continuous-space models (Kalchbrenner & Blun-som, 2013).”
Multilingual Models for Compositional Distributed Semantics (2014, background): “This results in the following objective function: (2) where θ is the set of all model variables.”
Deep Sequential Neural Network (2014, background): “The use of deep architectures have shown impressive results for many different tasks, from image classiﬁcation [10, 14], speech recognition [7] to machine translation [19] or even for natural language processing [16].”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “…of pivoting, that is, learning multiple bilingual spaces linked through the shared pivot English space is directly applicable to other prominent bilingual word embedding models such as (Chandar et al., 2014), (Hermann & Blunsom, 2014), (Soyer et al., 2015), (Zou et al., 2013), (Gouws et al., 2015).”
A Model of Coherence Based on Distributed Sentence Representation (2014, methodology): “,(Collobert and Weston, 2008; Zou et al., 2013)).”