Multilingual Distributed Representations without Word Alignment

By Karl Moritz Hermann, Phil Blunsom. 2013

161 citations recorded by the source corpus.

Distributed representations of meaning are a natural way to encode covariance relationships between words and phrases in NLP. By overcoming data sparsity problems, as well as providing information about semantic relatedness which is not available in discrete representations, distributed representations have proven useful in many NLP tasks. Recent work has shown how compositional semantic representations can successfully be applied to a number of monolingual applications such as sentiment analysis. At the same time, there has been some initial success in work on learning shared word-level representations across languages. We combine these two approaches by proposing a method for learning distributed representations in a multilingual setup. Our model learns to assign similar embeddings to aligned sentences and dissimilar ones to sentence which are not aligned while not requiring word alignments. We show that our representations are semantically informative and apply them to a cross-lingual document classification task where we outperform the previous state of the art. Further, by employing parallel corpora of multiple language pairs we find that our model learns representations that capture semantic relationships across languages for which no parallel data was used.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Sequence to Sequence Learning with Neural Networks (2014, background): “[12], whose model represents the inputs and outputs by feedforward networks, and map them to similar points in space.”
Neural Machine Translation by Jointly Learning to Align and Translate (2014, methodology): “, 2014a), with an encoder and a decoder for each language, or involve a language-specific encoder applied to each sentence whose outputs are then compared (Hermann and Blunsom, 2014).”
An efficient framework for learning sentence representations (2018, background): “Hermann & Blunsom (2013) consider a similar objective of minimizing the inner product between paired sentences in different languages.”
Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (2018, background): “…has, over the past ﬁve years, come to dominate the ﬁeld of machine learning as deep learning based approaches have been shown to outperform conventional techniques in domains such as image recognition [1], speech recognition [17], and automated machine translation of natural language [21,6].”
BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain (2017, background): “Deep networks have been found to significantly outperform previous machine learning techniques in a wide variety of domains, including image recognition [1], speech processing [2], machine translation [3], [4], and a number of games [5], [6]; the performance of these models even surpasses human”
Deep Learning for Answer Sentence Selection (2014, background): “At the same time, neural network-based distributional sent ence models have achieved successes in many natural language processing tasks such as sentiment an alysis [9, 13], paraphrase detection [17] and document classification [11, 10].”
BilBOWA: Fast Bilingual Distributed Representations without Word Alignments (2014, methodology): “Conceptually, this regularizer consists of minimizing a distance function between the representations ri learned for words wi in the two domains, weighted by how similar they are, i.e. Ω(θ1, θ2) = ∑ wi∈V1 ∑ wj∈V2 sim(wi, wj) · distance(ri, rj).”
Multilingual Models for Compositional Distributed Semantics (2014, methodology): “Hermann and Blunsom (2014) propose a large-margin learner for multilingual word representations, similar to the basic additive model proposed here, which, like the approaches above, relies on a bag-of-words model for sentence representations.”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “Similar to previous work (Hermann & Blunsom, 2013; Gouws et al., 2015), they use the mean of the word embeddings of a sentence as the sentence representation y and constrain these to be close together.”
Knowledge Distillation for Bilingual Dictionary Induction (2017, methodology): “Some bilingual word embedding methods such as (Blunsom and Hermann, 2014; Gouws et al., 2015) require sentence or word aligned data, which our approach does not require.”
Incorporating Side Information into Recurrent Neural Network Language Models (2016, methodology): “…embedding vector, e = 1 T t y t , both inspired by (Hermann and Blunsom, 2014a); bigram convolution with sum-pooling, e = P t tanh ( y t − 1 + y t ) (Hermann and Blunsom, 2014b); and RNN a recurrent neural network over the word sequence (Sutskever et al., 2014), using the ﬁnal hidden state(s) as…”
Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context (2017, methodology): “A similar idea of using English as a bridging language was used in the models proposed in (Hermann and Blunsom, 2014) and (Coulmance et al., 2015).”
Multimodal One-shot Learning of Speech and Images (2018, methodology): “We train models with a hinge loss for triplet pairs [14–16, 18], defined as:”
The Geometry of Distributed Representations for Better Alignment, Attenuated Bias, and Improved Interpretability (2020, methodology): “2 Aligning Embeddings Across Languages and Embeddings Word embeddings have been used to place word vectors from multiple languages in the same space [45, 65].”