Learning Joint Multilingual Sentence Representations with Neural Machine Translation

By Holger Schwenk, Matthijs Douze. 2017

218 citations recorded by the source corpus.

In this paper, we use the framework of neural machine translation to learn joint sentence representations across six very different languages. Our aim is that a representation which is independent of the language, is likely to capture the underlying semantics. We define a new cross-lingual similarity measure, compare up to 1.4M sentence representations and study the characteristics of close sentences. We provide experimental evidence that sentences that are close in embedding space are indeed semantically highly related, but often have quite different structure and syntax. These relations also hold when comparing sentences in different languages.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “While notable recent approaches have considered building a shared sentence encoder for multiple languages using publicly available parallel corpora (Johnson et al., 2016; Schwenk et al., 2017; España-Bonet et al., 2017), the lack of a large-scale, sentence-level semantic evaluation has limited their”
Filtering and Mining Parallel Data in a Joint Multilingual Space (2018, methodology): “Train Test Pair en other aligned en other en-de 400k 414k 9580 397k 414k en-fr 370k 272k 9086 373k 277k en-ru 558k 461k 14435 566k 457k en-zh 89k 95k 1899 90k 92k embeddings from (Schwenk and Douze, 2017) for ru and zh, which were trained on the UN corpus.”
Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings (2018, methodology): “Following recent trend, we use a sequence-tosequence encoder-decoder to learn our sentence embeddings (Schwenk and Douze, 2017; EspañaBonet et al., 2017; Hassan et al., 2018; Schwenk, 2018).”
A Corpus for Multilingual Document Classification in Eight Languages (2018, methodology): “In this paper, we evaluate a recently proposed technique to learn joint multilingual sentence representations (Schwenk and Douze, 2017).”
An Empirical Analysis of NMT-Derived Interlingual Embeddings and Their Use in Parallel Sentence Identification (2017, background): “Finally, in a concurrent work, [27] describe how joint multilingual sentence representations are learned with an NMT architecture with multiple encoders and/or decoders.”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, methodology): “…NMT model as a generative paraphraser.3 Semantic Similarity Parallel corpora in many language pairs have been used to produce fixed-size, multilingual sentence representations (Schwenk and Douze, 2017; Wieting et al., 2017; Artetxe and Schwenk, 2018; Wieting et al., 2019; Raganato et al., 2019).”
The TALP-UPC Machine Translation Systems for WMT19 News Translation Task: Pivoting Techniques for Low Resource MT (2019, background): “However, the latter either needs to use independent encoders and decoders (Schwenk and Douze, 2017; Firat et al., 2016; Escolano et al., 2019) or when using universal encoder and decoders (Johnson et al., 2017) needs to add a tag in the source input to let the system know to which language it is…”
Facebook AI’s WMT20 News Translation Task Submission (2020, unknown): “LASER (Schwenk and Douze, 2017) multilingual sentence embeddings.”
A Bilingual Generative Transformer for Semantic Sentence Embedding (2019, background): “For training models 4http://opus.nlpl.eu/OpenSubtitles.php 5https://www.statmt.org/wmt10/ training-giga-fren.tar 6We use LSTMs in our ablations. with Transformers, we use Adam (Kingma and Ba, 2014) with β1 = 0.9, β2 = 0.98, and = 10−8.”
Byte-based Neural Machine Translation (2017, background): “Recently, there have appeared works in multilingual word representations (Schwenk et al., 2017; Espa˜na-Bonet et al., 2017) Most multilingual works are at the level of words.”
Multimodal and Multilingual Embeddings for Large-Scale Speech Mining (2021, background): “The multilingual sentence representations can be learned with neural machine translation framework (España-Bonet et al., 2017; Schwenk and Douze, 2017; Artetxe and Schwenk, 2019; Kvapilíková et al., 2020).”
On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference (2018, background): “Finally, investigating how multilingual systems learn semantics can bring a new perspective to questions of universality of re-presentation (Schwenk and Douze, 2017).”
XQA: A Cross-lingual Open-domain Question Answering Dataset (2019, methodology): “Another line of research extracts multilingual representation from the context vector of NMT models that are trained on parallel data (Schwenk and Douze, 2017; Artetxe and Schwenk, 2018), which may be complementary to multilingual BERT.”
Bridging the domain gap in cross-lingual document classification (2019, background): “Recently, more sophisticated extensions have been proposed based on seq2seq training of cross-lingual sentence embeddings (Schwenk & Douze (2017); Artetxe & Schwenk (2018)) and contextual word embeddings pre-trained on masked language modeling, notably multilingual BERT (Devlin et al. (2018)) and…”
Simple and Effective Paraphrastic Similarity from Parallel Translations (2019, methodology): “…previous work for cross-lingual representations has focused on models based on encoders from neural machine translation (Espana-Bonet et al., 2017; Schwenk and Douze, 2017; Schwenk, 2018) or deep architectures using a contrastive loss (Gr´egoire and Langlais, 2018; Guo et al., 2018; Chidambaram…”
From Bilingual to Multilingual Neural Machine Translation by Incremental Training (2019, background): “Language independent architectures(Firat et al., 2016a,b; Schwenk and Douze, 2017) in which each language has its own encoder and some additional mechanism is added to produce shared representations, as averaging of the context vectors or sharing the attention mechanism.”
Improving Zero-shot Translation with Language-Independent Constraints (2019, background): “Motivated by the literature in sentence embeddings (Schwenk and Douze, 2017; Wang et al., 2017), we take the average over time of the encoder states.”
(Self-Attentive) Autoencoder-based Universal Language Representation for Machine Translation (2018, background): “topic has been mainly on evaluating if the NMT architecture of encoder-decoder with recurrent neural networks (RNNs), with or without attention mechanisms, is able to reach a universal language while training multiple languages (Johnson et al., 2016; Schwenk and Douze, 2017).”