An Autoencoder Approach to Learning Bilingual Word Representations

By A. P. Sarath Chandar, Stanislas Lauly, Hugo Larochelle, Mitesh M. Khapra, Balaraman Ravindran, Vikas Chandrakant Raykar, Amrita Saha. 2014

348 citations recorded by the source corpus.

Cross-language learning allows one to use training data from one language to build models for a different language. Many approaches to bilingual learning require that we have word-level alignment of sentences from parallel corpora. In this work we explore the use of autoencoder-based methods for cross-language learning of vectorial word representations that are coherent between two languages, while not relying on word-level alignments. We show that by simply learning to reconstruct the bag-of-words representations of aligned sentences, within and between languages, we can in fact learn high-quality representations and do without word alignments. We empirically investigate the success of our approach on the problem of cross-language text classification, where a classifier trained on a given language (e.g., English) must learn to generalize to a different language (e.g., German). In experiments on 3 language pairs, we show that our approach achieves state-of-the-art performance, outperforming a method exploiting word alignments and a strong machine translation baseline.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, methodology): “In (Chandar et al., 2014), a feedforward neural network was trained to learn a mapping from a bag-of-words representation of an input phrase to an output phrase.”
Learning bilingual word embeddings with (almost) no bilingual data (2017, background): “(2012) and Chandar A P et al. (2014). They both had coverage problems that made the results hard to compare, and, when considering the correlations for the word pairs in their vocabulary, their performance was poor. (7)Upadhyay et al. (2016) report similar problems using”
Offline bilingual word vectors, orthogonal transformations and the inverted softmax (2017, background): “Chandar et al. (2014) showed that such corpora can be used alongside monolingual text sources to learn online bilingual vectors, but to date, offline bilingual vectors have only been obtained from dictionaries.”
A Multiplicative Model for Learning Distributed Text-Based Attribute Representations (2014, methodology): “We are competitive with the current state-of-the-art approaches, being outperformed only by BiCVM+ [21] and BAE-corr [20] on EN → DE.”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “The same idea of pivoting, that is, learning multiple bilingual spaces linked through the shared pivot English space is directly applicable to other prominent bilingual word embedding models such as (Chandar et al., 2014), (Hermann & Blunsom, 2014), (Soyer et al.”
Siamese CBOW: Optimizing Word Embeddings for Sentence Representations (2016, background): “Many models related to the one we present here are used in a multilingual setting (Hermann and Blunsom, 2014b; Hermann and Blunsom, 2014a; Lauly et al., 2014).”
Ten Pairs to Tag – Multilingual POS Tagging via Coarse Mapping between Embeddings (2016, background): “Multilingual Word Embeddings There is an expansive body of research on learning multilingual word embeddings (Gouws et al., 2014; Faruqui and Dyer, 2014; Lu et al., 2015; Lauly et al., 2014; Luong et al., 2015).”
Collaborative Denoising Auto-Encoders for Top-N Recommender Systems (2016, neutral): “An alternative solution is to build a Hierarchical Softmax tree on the output layer [10], but it requires the loss function on the output layer to be softmax loss.”
The representational geometry of word meanings acquired by neural machine translation models (2017, methodology): “Various studies have demonstrated that word representations can also be effectively learned from bilingual corpora, aligned at the document, paragraph or word level (Haghighi et al. 2008; Vulić et al. 2011; Mikolov et al. 2013a; Hermann and Blunsom 2014; Chandar et al. 2014).”
On Deep Multi-View Representation Learning (2015, methodology): “The denominator of the discrepancy term is used to keep the optimization from improving the objective by simply scaling down the projections (although they can never become identically zero due to the reconstruction terms).”
Text Classification Algorithms: A Survey (2019, background): “The encoder representation involves a sum of the representation of all words (for bag-of-words), reflecting the relative frequency of each word [95]:”
Domain-Invariant Projection Learning for Zero-Shot Recognition (2018, background): “This is motivated by the fact that adding an autoencoder style self-reconstruction task can improve the model generalization ability as demonstrated in many other problems [30, 3].”