An Empirical Analysis of NMT-Derived Interlingual Embeddings and Their Use in Parallel Sentence Identification

By Cristina España-Bonet, Á. Varga, Alberto Barrón-Cedeño, Josef van Genabith. 2017

75 citations recorded by the source corpus.

End-to-end neural machine translation has overtaken statistical machine translation in terms of translation quality for some language pairs, specially those with large amounts of parallel data. Besides this palpable improvement, neural networks provide several new properties. A single system can be trained to translate between many languages at almost no additional cost other than training time. Furthermore, internal representations learned by the network serve as a new semantic representation of words—or sentences—which, unlike standard word embeddings, are learned in an essentially bilingual or even multilingual context. In view of these properties, the contribution of the present paper is twofold. First, we systematically study the neural machine translation (NMT) context vectors, i.e., output of the encoder, and their power as an interlingua representation of a sentence. We assess their quality and effectiveness by measuring similarities across translations, as well as semantically related and semantically unrelated sentence pairs. Second, as extrinsic evaluation of the first point, we identify parallel sentences in comparable corpora, obtaining an <inline-formula><tex-math notation="LaTeX">$F_1=98.2\%$</tex-math></inline-formula> on data from a shared task when using only NMT context vectors. Using context vectors jointly with similarity measures <inline-formula> <tex-math notation="LaTeX">$F_1$</tex-math></inline-formula> reaches <inline-formula><tex-math notation="LaTeX"> $98.9\%$</tex-math></inline-formula>.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “While notable recent approaches have considered building a shared sentence encoder for multiple languages using publicly available parallel corpora (Johnson et al., 2016; Schwenk et al., 2017; España-Bonet et al., 2017), the lack of a large-scale, sentence-level semantic evaluation has limited their”
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia (2019, methodology): “Closest to this approach is the research described in España-Bonet et al. (2017); Hassan et al. (2018); Guo et al. (2018); Yang et al. (2019).”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, methodology): “Closest to this approach is the research described in Espa˜na-Bonet et al. (2017); Hassan et al. (2018); Guo et al. (2018); Yang et al. (2019).”
Multimodal and Multilingual Embeddings for Large-Scale Speech Mining (2021, background): “The multilingual sentence representations can be learned with neural machine translation framework (España-Bonet et al., 2017; Schwenk and Douze, 2017; Artetxe and Schwenk, 2019; Kvapilíková et al., 2020).”
XQA: A Cross-lingual Open-domain Question Answering Dataset (2019, methodology): “To address this issue, much effort has been devoted to using the context vector of NMT system as multilingual sentence representation (Schwenk and Douze, 2017; Espana-Bonet et al., 2017).”
Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining (2020, background): “Similarly, Schwenk and Douze (2017); Schwenk (2018); Espana-Bonet et al. (2017) derive sentence embeddings from internal representations of a neural machine translation system with a shared encoder.”
XCMRC: Evaluating Cross-lingual Machine Reading Comprehension (2019, background): “Conneau (2017) and Espa (2017) jointly trained a sequence to sequence MT system on multiple languages to learn a shared multilingual sentence embedding space [4] [9].”
Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation (2020, background): “…sentence representations obtained from NMT systems or tailored architectures are achieving a new state-of-the-art in parallel sentence extraction and ﬁltering (Espa˜na-Bonet et al., 2017; Gr´egoire and Langlais, 2018; Artetxe and Schwenk, 2019; Hangya and Fraser, 2019; Chaudhary et al., 2019).”
Towards Interlingua Neural Machine Translation (2019, background): “Schwenk et al. (2017) and Espana-Bonet et al. (2017) evaluate how a recurrent NMT architecture without attention is able to generate a common representation between languages.”
(Self-Attentive) Autoencoder-based Universal Language Representation for Machine Translation (2018, background): “Schwenk et al. (2017) and Espana-Bonet et al. (2017) evaluate how a recurrent NMT architecture without attention is able to generate a common representation between languages.”
A Bilingual Generative Transformer for Semantic Sentence Embedding (2019, background): “We also implicitly compare to previous machine translation approaches like Espana-Bonet et al. (2017); Schwenk and Douze (2017); Artetxe and Schwenk (2018) in Appendix A where we explore different variations of training LSTM sequenceto-sequence models.”