BilBOWA: Fast Bilingual Distributed Representations without Word Alignments

By Stephan Gouws, Yoshua Bengio, Greg S. Corrado. 2014

396 citations recorded by the source corpus.

We introduce BilBOWA (Bilingual Bag-of-Words without Alignments), a simple and computationally-efficient model for learning bilingual distributed representations of words which can scale to large monolingual datasets and does not require word-aligned parallel training data. Instead it trains directly on monolingual data and extracts a bilingual signal from a smaller set of raw-text sentence-aligned data. This is achieved using a novel sampled bag-of-words cross-lingual objective, which is used to regularize two noise-contrastive language models for efficient cross-lingual feature learning. We show that bilingual embeddings learned using the proposed model outperform state-of-the-art methods on a cross-lingual document classification task as well as a lexical translation task on WMT11 data.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “The first approach is commonly used as a strong baseline for monolingual sentence embeddings (Arora et al., 2017; Conneau and Kiela, 2018; Gouews et al., 2014).”
Unsupervised Neural Machine Translation (2017, methodology): “Most methods for learning cross-lingual word embeddings rely on some bilingual signal at the document level, typically in the form of parallel corpora (Gouws et al., 2015; Luong et al., 2015a).”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “Cross-lingual representations Early work focused on learning cross-lingual representations using either parallel corpora (Gouws et al., 2015; Luong et al., 2015) or a bilingual dictionary to learn a linear transformation (Mikolov et al.”
Word Translation Without Parallel Data (2017, background): “Most successful approaches for learning cross-lingual word embeddings have used bilingual dictionaries to train the mapping from the source to the target space Mikolov et al. (2013b); Zou et al. (2013); Faruqui & Dyer (2014); Ammar et al.”
Learning protein sequence embeddings using information from structure (2019, methodology): “We first demonstrate the importance of our SSA mechanism when training the contextual embedding model by comparing the performance of biLSTM encoders trained with SSA versus the same encoders trained with uniform alignment and a mean embedding comparison approaches [22].”
Learning bilingual word embeddings with (almost) no bilingual data (2017, methodology): “Most methods to learn these multilingual word embeddings make use of large parallel corpora (Gouws et al., 2015; Luong et al., 2015), but there have been several proposals to relax this requirement, given its scarcity in most language pairs.”
Offline bilingual word vectors, orthogonal transformations and the inverted softmax (2017, background): “However our performance appears competitive with Bilbowa, a leading method for learning bilingual vectors online from monolingual corpora and aligned text (Gouws et al., 2015).”
Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation (2018, background): “BilBOWA (Gouws, Bengio, and Corrado 2015) is a simpler model which extends skip-gram with negative sampling (Mikolov et al. 2013) to optimize each word’s similarity with its context in both the current language and the other parallel language.”
Massively Multilingual Word Embeddings (2016, background): “…et al., 2013; Hermann and Blunsom, 2014; Cho et al., 2014; Luong et al., 2015b; Luong et al., 2015a, inter alia ), cross-lingual dependency parsing (Guo et al., 2015; Guo et al., 2016), and cross-lingual document classiﬁcation (Klementiev et al., 2012; Gouws et al., 2014; Kocisk`y et al., 2014).”
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer (2020, background): “…focused on learning general-purpose cross-lingual representations that can be applied to many tasks, ﬁrst on the word level (Mikolov et al., 2013; Gouws et al., 2015; Glavaˇs et al., 2019; Ruder et al., 2019; Wang et al., 2020) and later on the full-sentence level (Chidambaram et al., 2019;…”
A Primer on Neural Network Models for Natural Language Processing (2015, methodology): “For further discussion on the LSTM architecture see the PhD thesis by Alex Graves (2008), as well as the online-post by Olah (2015b). For an analysis of the behavior of an LSTM when used as a character-level language model, see the work of Karpathy et al. (2015). For further explanation of the motiv”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “…of pivoting, that is, learning multiple bilingual spaces linked through the shared pivot English space is directly applicable to other prominent bilingual word embedding models such as (Chandar et al., 2014), (Hermann & Blunsom, 2014), (Soyer et al., 2015), (Zou et al., 2013), (Gouws et al., 2015).”
Overview of the Second BUCC Shared Task: Spotting Parallel Sentences in Comparable Corpora (2017, methodology): “They trained bilingual word embeddings with BilBOWA (Bilingual Bag-of-Words without Alignments (Gouws et al., 2015)) on the Europarl paral-”
Learning Joint Multilingual Sentence Representations with Neural Machine Translation (2017, methodology): “Without using word alignments, BilBOWA (Gouews et al., 2014) optimizes both monolingual and bilingual objectives, uses Skip-gram as monolingual loss, while formulating the bilingual loss as Euclidean distance between bag-ofwords representations of aligned sentences.”
Multilingual Hierarchical Attention Networks for Document Classification (2017, background): “The goal is to learn an aligned word embedding space for multiple languages by leveraging bilingual dictionaries (Faruqui and Dyer, 2014; Ammar et al., 2016), parallel sentences (Gouws et al., 2015) or comparable documents such as Wikipedia pages (Yih et al., 2011; Al-Rfou et al., 2013).”
Cross-lingual Models of Word Embeddings: An Empirical Comparison (2016, methodology): “Some methods do not have publicly available code (Coulmance et al., 2015; Zou et al., 2013); for others, like BilBOWA (Gouws et al., 2015), we identiﬁed problems in the available code, which caused it to consistently produced results that are inferior even to mono-lingually trained vectors.”
Ten Pairs to Tag – Multilingual POS Tagging via Coarse Mapping between Embeddings (2016, background): “Multilingual Word Embeddings There is an expansive body of research on learning multilingual word embeddings (Gouws et al., 2014; Faruqui and Dyer, 2014; Lu et al., 2015; Lauly et al., 2014; Luong et al., 2015).”
Bilingual Word Embeddings from Parallel and Non-parallel Corpora for Cross-Language Text Classification (2016, methodology): “To compute the bilingual regularization function ϕ ( · ) , we slightly deviate from earlier approaches (Gouws et al., 2015).”