Effective Parallel Corpus Mining using Bilingual Sentence Embeddings

By Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Matthew Cer, Gustavo Hernández Ábrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope and 1 more. 2018

115 citations recorded by the source corpus.

This paper presents an effective approach for parallel corpus mining using bilingual sentence embeddings. Our embedding models are trained to produce similar representations exclusively for bilingual sentence pairs that are translations of each other. This is achieved using a novel training method that introduces hard negatives consisting of sentences that are not translations but have some degree of semantic similarity. The quality of the resulting embeddings are evaluated on parallel corpus reconstruction and by assessing machine translation systems trained on gold vs. mined sentence pairs. We find that the sentence embeddings can be used to reconstruct the United Nations Parallel Corpus (Ziemski et al., 2016) at the sentence-level with a precision of 48.9% for en-fr and 54.9% for en-es. When adapted to document-level matching, we achieve a parallel document matching accuracy that is comparable to the significantly more computationally intensive approach of Uszkoreit et al. (2010). Using reconstructed parallel data, we are able to train NMT models that perform nearly as well as models trained on the original data (within 1-2 BLEU).

Subfields: Artificial Intelligence, Molecular Biology, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, background): “Compared to supervised approaches such as [19, 49], CRISS performs mining with unsupervised sentence representations pretrained from large monolingual data.”
Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings (2018, methodology): “This issue was also pointed out by Guo et al. (2018), who learn an encoder to score known translation pairs above synthetic negative examples and train a separate model to dynamically scale and shift the dot product on held out supervised data.”
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks (2019, background): “So we introduce the hard negative samples followed Guo et al. (2018).”
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia (2019, methodology): “Closest to this approach is the research described in España-Bonet et al. (2017); Hassan et al. (2018); Guo et al. (2018); Yang et al. (2019).”
ParaCrawl: Web-Scale Acquisition of Parallel Corpora (2020, methodology): “Bilingual sentence embeddings (Guo et al., 2018) and multilingual sentence embeddings (Artetxe and Schwenk, 2018) were tested on their ability to reconstruct parallel corpora.”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, methodology): “Closest to this approach is the research described in Espa˜na-Bonet et al. (2017); Hassan et al. (2018); Guo et al. (2018); Yang et al. (2019).”
Non-Parametric Adaptation for Neural Machine Translation (2019, background): “Retrieving information from unsupervised corpora by utilizing multilingual representations (Guo et al., 2018) might be another interesting extension of this work.”
Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax (2019, methodology): “Following [ Guo et al. , 2018 ] , we also integrate semantically similar hard-negative examples in training.”
Multimodal and Multilingual Embeddings for Large-Scale Speech Mining (2021, background): “It has been observed that an absolute threshold on the cosine distance is globally inconsistent (Guo et al., 2018).”
Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions (2019, background): “Guo et al. (2018) leveraged hard negatives to correctly identify translation pairs.”
All You May Need for VQA are Image Captions (2022, background): “…a complex multimodal task that, to be successfully modeled and evaluated, requires large amounts of annotations that are not naturally produced by existing business processes, the way translation-pair annotations (Guo et al., 2018) or image alt-text annotations (Sharma et al., 2018) are produced.”
Simple and Effective Paraphrastic Similarity from Parallel Translations (2019, methodology): “…for cross-lingual representations has focused on models based on encoders from neural machine translation (Espana-Bonet et al., 2017; Schwenk and Douze, 2017; Schwenk, 2018) or deep architectures using a contrastive loss (Gr´egoire and Langlais, 2018; Guo et al., 2018; Chidambaram et al., 2018).”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “These include bilingual dual encoder architectures [71, 201], and shared multilingual encoder-decoder architectures [161].”
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, methodology): “In unimodal text-only setups, Transformer-based architectures have recently been integrated with bi-encoder (BE) methods (Guo et al., 2018; Reimers and Gurevych, 2019; Humeau et al., 2020; Hen-derson et al., 2020; Feng et al., 2020, inter alia ), yielding computationally more efﬁcient sentence…”
Language-agnostic BERT Sentence Embedding (2020, methodology): “Cross-lingual embedding models trained with inbatch negative samples benefits from large training batch sizes (Guo et al., 2018).”
ConveRT: Efficient and Accurate Conversational Representations from Transformers (2019, background): “, 2017), bilingual text mining and representation learning (Guo et al., 2018; Chidambaram et al., 2019), and question answering (Humeau et al.”
Parallel Sentence Mining by Constrained Decoding (2020, background): “Guo et al. (2018) design a dual encoder model to learn multilingual sentence embeddings directly with added negative examples.”
ReQA: An Evaluation for End-to-End Answer Retrieval Models (2019, methodology): “, 2018), translation pair retrieval (Guo et al., 2018; Yang et al., 2019b) and similar text retrieval (Gillick et al.”