Learning Distributed Representations for Multilingual Text Sequences

By Hieu Pham, Thang Luong, Christopher D. Manning. 2015

63 citations recorded by the source corpus.

We propose a novel approach to learning distributed representations of variable-length text sequences in multiple languages simultaneously. Unlike previous work which often derive representations of multi-word sequences as weighted sums of individual word vectors, our model learns distributed representations for phrases and sentences as a whole. Our work is similar in spirit to the recent paragraph vector approach but extends to the bilingual context so as to efficiently encode meaning-equivalent text sequences of multiple languages in the same semantic space. Our learned embeddings achieve state-of-theart performance in the often used crosslingual document classification task (CLDC) with an accuracy of 92.7 for English to German and 91.5 for German to English. By learning text sequence representations as a whole, our model performs equally well in both classification directions in the CLDC task in which past work did not achieve.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “Pham et al. (2015) directly train embedding representations for sentences with no attempt at compositionality.”
Filtering and Mining Parallel Data in a Joint Multilingual Space (2018, background): “There are several works on learning multilingual sentence embeddings which could be used for that purpose, i.e. (Hermann and Blunsom, 2014; Pham et al., 2015; Zhou et al., 2016; Chan-dar et al., 2013; Mogadala and Rettinger, 2016).”
A Survey of Cross-lingual Word Embedding Models (2017, background): “Extensions of sentence-alignment models Mogadala and Rettinger (2016) extend the approach of Pham et al. (2015) to also work without parallel data and adjust the regularization term Ω based on the nature of the training corpus.”
Learning Joint Multilingual Sentence Representations with Neural Machine Translation (2017, background): “There are several works on learning multilingual representations at document level (Hermann and Blunsom, 2014; Zhou et al., 2016b; Pham et al., 2015).”
Cross-Lingual Sentiment Classification with Bilingual Document Representation Learning (2016, background): “Pham et al. (2015) extend the paragraph vector model to force bilingual sentences to share the same sentence vector.”
MultiVec: a Multilingual and Multilevel Representation Learning Toolkit for NLP (2016, background): “, 2015], or bilingual representations for text sequences [Pham et al., 2015].”
Bilingual Word Embeddings from Parallel and Non-parallel Corpora for Cross-Language Text Classification (2016, methodology): “Also, word representations were extended to meet larger textual units like phrases, sentences and documents either monolingual (Socher et al., 2012; Le and Mikolov, 2014) or bilingual (Pham et al., 2015).”
Attention-based LSTM Network for Cross-Lingual Sentiment Classification (2016, background): “Pham et al. (2015) extended the paragraph model into bilingual setting.”
Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations (2018, background): “Several authors proposed to extend ParagraphVec (Le and Mikolov, 2014) to the cross-lingual case: Pham et al. (2015) add a bilingual constraint to learn cross-lingual representations using aligned sentences; Mogadala and Rettinger (2016) add a general cross-lingual regularization term to…”