Learning Bilingual Word Representations by Marginalizing Alignments

By Tomás Kociský, Karl Moritz Hermann, Phil Blunsom. 2014

83 citations recorded by the source corpus.

We present a probabilistic model that simultaneously learns alignments and distributed representations for bilingual data. By marginalizing over word alignments the model captures a larger semantic context than prior work relying on hard alignments. The advantage of this approach is demonstrated in a cross-lingual classification task, where we outperform the prior published state of the art.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “Many of these methods require some form of supervision (typically in the form of a small bilingual lexicon) to align two sets of source and target embeddings to the same space (Mikolov et al., 2013a; Kociský et al., 2014; Faruqui and Dyer, 2014; Ammar et al., 2016).”
Learning protein sequence embeddings using information from structure (2019, methodology): “[20] learn bilingual word embeddings jointly with a FastAlign [21] word alignment model using expectation maximization.”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “The complete objective is the following: J = L(Xs) + L(Xt) + Ω(As→t,Xs) + Ω(At→s,Xt) (27) Joint learning of word embeddings and word alignments Kočiský, Hermann, and Blunsom (2014) simultaneously learn word embeddings and word-level alignments using a distributed version of FastAlign (Dyer,…”
Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation (2015, methodology): “(Blunsom et al., 2014) proposed a novel approach based on a joint optimization method for word alignments and the embedding.”
Learning Joint Multilingual Sentence Representations with Neural Machine Translation (2017, methodology): “DWA (Distributed Word Alignment) (Kociský et al., 2014) learns word alignments and bilingual word embeddings simultaneously using translation probability as objective.”
Earth Mover’s Distance Minimization for Unsupervised Bilingual Lexicon Induction (2017, background): “…data, at the document level (Vuli´c and Moens, 2015), the sentence level (Zou et al., 2013; Chandar A P et al., 2014; Hermann and Blunsom, 2014; Koˇcisk´y et al., 2014; Gouws et al., 2015; Luong et al., 2015; Coul-mance et al., 2015; Oshikiri et al., 2016), or the word level (i.e. in the form…”
The representational geometry of word meanings acquired by neural machine translation models (2017, methodology): “…2011;Mikolov et al. 2013a) and document 1 Subsequent variants use different algorithms for selecting the (w, c) from the training corpus (Hill and Korhonen 2014; Levy and Goldberg 2014). classification (Klementiev et al. 2012a; Hermann and Blunsom 2014; Chandar et al. 2014; Kočiský et al. 2014).”
Knowledge Distillation for Bilingual Dictionary Induction (2017, background): “, 2012), information retrieval (Vulic and Moens, 2015), and cross-lingual document classification (Klementiev et al., 2012; Kočiskỳ et al., 2014).”