Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings

By Mikel Artetxe, Holger Schwenk. 2018

221 citations recorded by the source corpus.

Machine translation is highly sensitive to the size and quality of the training data, which has led to an increasing interest in collecting and filtering large parallel corpora. In this paper, we propose a new method for this task based on multilingual sentence embeddings. In contrast to previous approaches, which rely on nearest neighbor retrieval with a hard threshold over cosine similarity, our proposed method accounts for the scale inconsistencies of this measure, considering the margin between a given sentence pair and its closest candidates instead. Our experiments show large improvements over existing methods. We outperform the best published results on the BUCC mining task and the UN reconstruction task by more than 10 F1 and 30 precision points, respectively. Filtering the English-German ParaCrawl corpus with our approach, we obtain 31.2 BLEU points on newstest2014, an improvement of more than one point over the best official filtered version.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, methodology): “We focus on the embeddings generated by the LASER encoder, which enables the comparison of sentences in 94 different languages (Artetxe and Schwenk, 2018a).”
MLQA: Evaluating Cross-lingual Extractive Question Answering (2019, methodology): “To detect parallel sentences we use the LASER toolkit, 3 which achieves state-of-the-art performance in mining parallel sentences (Artetxe and Schwenk, 2019).”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, methodology): “To mine input output pairs, we follow previous work on unsupervised machine translation and employ the margin function formulation (Artetxe and Schwenk, 2019; Tran et al., 2020) based on K nearest neighbors (KNN).”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, methodology): “…vector representation of two sentences in languages i and j respectively, we score the semantic similarity of x and y using a ratio margin function [1] deﬁned as the following: score ( x, y ) = cos ( x, y ) where N x is the KNN neighborhood of x in the monolingual dataset of y ’s language; and N y…”
Parallel Corpus Filtering via Pre-trained Language Models (2020, background): “…on parallel corpus ﬁltering performs poorly in our scenario as it either requires large clean parallel corpora or dictionaries (Xu and Koehn, 2017; Artetxe and Schwenk, 2019; Junczys-Dowmunt, 2018; Chaudhary et al., 2019), or relies on multilingual word embeddings and ne-glects context when…”
Unsupervised Question Decomposition for Question Answering (2020, methodology): “Thus, we take inspiration from work in machine translation on parallel corpus mining (Xu & Koehn, 2017; Artetxe & Schwenk, 2019) and in unsupervised QA (Lewis et al., 2019).”
LongForm: Effective Instruction Tuning with Reverse Instructions (2023, neutral): “Thus, our dataset features a diverse collection of instructions (including human-written, template-based and LLM-generated), each paired with a human-written output.”
ParaCrawl: Web-Scale Acquisition of Parallel Corpora (2020, background): “, 2018) and multilingual sentence embeddings (Artetxe and Schwenk, 2018) were tested on their ability to reconstruct parallel corpora.”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, methodology): “In this work, we rely on massively multilingual sentence embeddings and margin-based mining in the joint embedding space, as described in (Schwenk, 2018; Artetxe and Schwenk, 2018a,b).”
Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings (2019, methodology): “The LASER model (Artetxe and Schwenk, 2018a) makes use of multilingual sentence representations to gauge the similarity between the source and the target sentence.”
MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases (2022, unknown): “We compute LASER embed-dings of dimension 1024 and reduce dimensionality with a 512 PCA followed by random rotation.”
Multimodal and Multilingual Embeddings for Large-Scale Speech Mining (2021, methodology): “In this work, we build on recent advances to perform bitext mining based on a similarity measure in a ﬁxed-size sentence embedding space (Schwenk, 2018; Artetxe and Schwenk, 2019; Feng et al., 2020).”
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, background): “Similarity search is frequently used to evaluate multilingual text encoders (Artetxe and Schwenk, 2018; Feng et al., 2020; Heffernan et al., 2022).”