Multilingual Models for Compositional Distributed Semantics

By Karl Moritz Hermann, Phil Blunsom. 2014

319 citations recorded by the source corpus.

We present a novel technique for learning semantic representations, which extends the distributional hypothesis to multilingual data and joint-space embeddings. Our models leverage parallel data and learn to strongly align the embeddings of semantically equivalent sentences, while maintaining sufficient distance between those of dissimilar sentences. The models do not rely on word alignments or any syntactic information and are successfully applied to a number of diverse languages. We extend our approach to learn semantic representations at the document level, too. We evaluate these models on two cross-lingual document classification tasks, outperforming the prior state of the art. Through qualitative analysis and the study of pivoting effects we demonstrate that our representations are semantically plausible and can capture semantic relationships across languages without parallel data.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Cross-lingual Language Model Pretraining (2019, methodology): “By using parallel data, Hermann and Blunsom (2014); Conneau et al. (2018b); Eriguchi et al. (2018) investigated zero-shot cross-lingual sentence classi-ﬁcation.”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “Hermann and Blunsom (2014) propose a compositional vector model involving unigrams”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “This is analogous to pairwise ranking methods used in machine translation [13, 14].”
Filtering and Mining Parallel Data in a Joint Multilingual Space (2018, background): “There are several works on learning multilingual sentence embeddings which could be used for that purpose, i.e. (Hermann and Blunsom, 2014; Pham et al., 2015; Zhou et al., 2016; Chan-dar et al., 2013; Mogadala and Rettinger, 2016).”
Towards Universal Paraphrastic Sentence Embeddings (2015, methodology): “We tune over the strategy in our experiments.”
Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation (2018, background): “BiCVM (Hermann and Blunsom 2014) learns bilingual compositional representations from sentence-aligned parallel corpora.”
Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval (2015, methodology): “In [18], there are two main composition models, ADD model that is bag of words and BI model that is a summation over bi-gram pairs plus a non-linearity.”
Massively Multilingual Word Embeddings (2016, background): “There is a rich body of literature on bilingual embeddings, including work on machine translation (Zou et al., 2013; Hermann and Blunsom, 2014; Cho et al., 2014; Luong et al., 2015b; Luong et al., 2015a, inter alia ), cross-lingual dependency parsing (Guo et al., 2015; Guo et al., 2016), and…”
A Primer on Neural Network Models for Natural Language Processing (2015, background): “Many problems in NLP involve structured outputs: cases where the desired output is not a class label or distribution over class labels, but a structured object such as a sequence, a tree or a graph.”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “Hermann and Blunsom (2014) extend this approach to documents, by applying the composition and objective function recursively to compose sentences into documents.”
Monolingual and Cross-Lingual Information Retrieval Models Based on (Bilingual) Word Embeddings (2015, methodology): “Moreover, following the recent trend of multilingual word embedding induction (e.g., [14, 11]), in this paper we also focus on the induction of bilingual word embeddings (BWEs), and show how to use BWEs in cross-lingual information retrieval tasks .”
Aligning Cross-Lingual Entities with Multi-Aspect Information (2019, background): “Recent works (Hermann and Blunsom, 2014; Conneau et al., 2018; Eriguchi et al., 2018) studied cross-lingual sentence classiﬁcation via zero-shot learning.”
Transfer learning for speech and language processing (2015, methodology): “As in [139], only sentence pairs are required in the learning; the difference is that the embedding leveraged both monolingual data and bi-lingual data, and employd noise-contrastive training to improve efficiency.”
User Modeling with Neural Network for Review Rating Prediction (2015, methodology): “this paper, we exploit a simple and effective approach [Hermann and Blunsom, 2014], which recursively uses biTanh function to produce the document representation.”