Bilingual Word Representations with Monolingual Quality in Mind

By Thang Luong, Hieu Pham, Christopher D. Manning. VS@HLT-NAACL, 2015

340 citations recorded by the source corpus.

Recent work in learning bilingual representations tend to tailor towards achieving good performance on bilingual tasks, most often the crosslingual document classification (CLDC) evaluation, but to the detriment of preserving clustering structures of word representations monolingually. In this work, we propose a joint model to learn word representations from scratch that utilizes both the context coocurrence information through the monolingual component and the meaning equivalent signals from the bilingual constraint. Specifically, we extend the recently popular skipgram model to learn high quality bilingual representations efficiently. Our learned embeddings achieve a new state-of-the-art accuracy of 80.3 for the German to English CLDC task and a highly competitive performance of 90.7 for the other classification direction. At the same time, our models outperform best embeddings from past bilingual representation work by a large margin in the monolingual word similarity evaluation. 1

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c7188396767227c0d9ed087a7b077f22fccd7372

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond (2018, background): “Most of this research focuses on cross-lingual word embeddings (Ruder et al., 2017), which are commonly learned jointly from parallel corpora (Gouws et al., 2015; Luong et al., 2015).”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “Cross-lingual representations Early work focused on learning cross-lingual representations using either parallel corpora (Gouws et al., 2015; Luong et al., 2015) or a bilingual dictionary to learn a linear transformation (Mikolov et al.”
The Geometry of Distributed Representations for Better Alignment, Attenuated Bias, and Improved Interpretability (2020, background): “[70] J. Pennington, R. Socher, and C. Manning.”
Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation (2017, methodology): “…compared to all other models in our comparison. paragram-300d , (Wieting et al., 2015)) further re-ﬁned using linguistic constraints ( paragram+cf-300d , (Mrkˇsi´c et al., 2016)); (3) Multilingual embedding models from Luong et al. (2015) ( biskip-256d ) and Faruqui and Dyer (2014) ( bicca-512d ).”
Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation (2018, background): “Luong, Pham, and Manning (2015) also proposed obtaining cross-lingual representations using a similar approach.”
Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints (2017, background): “…signal/supervision they use to tie languages into uniﬁed bilingual vector spaces: some models learn on the basis of parallel word-aligned data (Luong et al., 2015; Coulmance et al., 2015) or sentence-aligned data (Hermann and Blunsom, 2014a; Hermann and Blunsom, 2014b; Chandar et al., 2014;…”
Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations (2018, background): “Other approaches predict mono-and bilingual context using word alignment information as an extension to the standard skip-gram model (Luong et al., 2015) or inject cross-lingual synonymy and antonymy constraints similar as in the monolingual setting (Mrkˇsi´c et al., 2017).”
Learning principled bilingual mappings of word embeddings while preserving monolingual invariance (2016, background): “Bilingual word embeddings have attracted a lot of attention in recent times (Zou et al., 2013; Kočiský et al., 2014; Chandar A P et al., 2014; Gouws et al., 2014; Gouws and Søgaard, 2015; Luong et al., 2015; Wick et al., 2016).”
Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context (2017, methodology): “This modeling approach is reminiscent of (Luong et al., 2015), who jointly learned embeddings for two languages l 1 and l 2 by optimizing a joint objective containing 4 skip-gram terms using the aligned pair ( x e , x f )– two predicting monolingual contexts Learning.”
Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classification (2016, methodology): “Since many of these methods require a large amount of parallel corpus or extended training time, in this work we leverage the pre-trained bilingual word embeddings (BWE) in (Zou et al., 2013).”
Learning Joint Multilingual Sentence Representations with Neural Machine Translation (2017, background): “…handle several source and/or target languages at once, with the goal of achieving better translation quality than with separately trained NMT systems, in particular for under resourced languages, see for instance (Dong et al., 2015; Zoph and Knight, 2016; Luong et al., 2015a; Firat et al., 2016a).”
Cross-Lingual Sentiment Classification with Bilingual Document Representation Learning (2016, background): “Meng et al. (2012) and Lu et al.”
XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering (2019, background): “Some instead train multilingual word embeddings jointly from parallel corpora (Gouws et al., 2015; Luong et al., 2015).”
Bilingual Word Embeddings from Parallel and Non-parallel Corpora for Cross-Language Text Classification (2016, background): “…to multilingual (Hermann and Blunsom, 2014; Koˇcisk´y et al., 2014; Coulmance et al., 2015), bilingual (Gouws et al., 2015; Vuli´c and Moens, 2015; Luong et al., 2015) and polylin-gual (Al-Rfou et al., 2013) settings by projecting multiple or pair of languages into the shared semantic space.”
Pre-trained models for natural language processing: A survey (2020, background): “tions shared across languages plays an important role in many cross-lingual NLP tasks. Cross-Lingual Language Understanding (XLU) Most of the early works focus on learning multilingual word embedding [45, 123, 158], which represents text from multiple languages in a single semantic space. However, t”
Cross-lingual Models of Word Embeddings: An Empirical Comparison (2016, methodology): “These tasks have been used in previous works (Klementiev et al., 2012; Luong et al., 2015; Vuli´c and Moens, 2013a; Guo et al., 2015) for evaluating cross-lingual embeddings, but no comparison exists which uses them in conjunction.”
H2@BUCC18: Parallel Sentence Extraction from Comparable Corpora Using Multilingual Sentence Embeddings (2018, methodology): “Much research work has been conducted to address this problem, following several approaches (Luong et al., 2015): (i) Bilingual mapping, where word representations are trained for each language independently, and a linear mapping is then learned to transform representations from one language to…”