Filtering and Mining Parallel Data in a Joint Multilingual Space

By Holger Schwenk. 2018

114 citations recorded by the source corpus.

We learn a joint multilingual sentence embedding and use the distance between sentences in different languages to filter noisy parallel data and to mine for parallel data in large news collections. We are able to improve a competitive baseline on the WMT’14 English to German task by 0.3 BLEU by filtering out 25% of the training data. The same approach is used to mine additional bitexts for the WMT’14 system and to obtain competitive results on the BUCC shared task to identify parallel sentences in comparable corpora. The approach is generic, it can be applied to many language pairs and it is independent of the architecture of the machine translation system.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Discovering Language Model Behaviors with Model-Written Evaluations (2022, methodology): “This method has seen success in machine translation (Schwenk, 2018; Zhang et al., 2020; Tran et al., 2020), text classification (Du et al.”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, background): “There is also a large area of research in using sentence-level representations to mine pseudo-parallel sentence pairs [6, 13, 19, 22, 48, 49, 52].”
Parallel Corpus Filtering via Pre-trained Language Models (2020, background): “Chaudhary et al. (2019) train a multilingual sentence encoder on various English-Foreign Language parallel corpus and prove the zero-shot cross-lingual transfer capability between non-English pairs, such as Japanese and Chinese.”
Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings (2018, methodology): “8 Our results show that multilingual sentence embeddings already achieve competitive performance using standard forward retrieval over cosine similarity, which is in line with Schwenk (2018).”
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia (2019, methodology): “In this work, we build on a recent approach to mine parallel texts based on a distance measure in a joint multilingual sentence embedding space (Schwenk, 2018; Artetxe and Schwenk, 2018a).”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, methodology): “In this work, we rely on massively multilingual sentence embeddings and margin-based mining in the joint embedding space, as described in (Schwenk, 2018; Artetxe and Schwenk, 2018a,b).”
A Massive Collection of Cross-Lingual Web-Document Pairs (2019, methodology): “We use the open-source LASER toolkit (Schwenk, 2018) with the marginbased filtering criterion to mine sentences, as”
Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings (2019, methodology): “The LASER model (Artetxe and Schwenk, 2018a) makes use of multilingual sentence representations to gauge the similarity between the source and the target sentence.”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, methodology): “We find that the probability of sys as estimated by an LM, as well as and the cosine distance between LASER embeddings of sys and ref , both have decent correlation with human judgments and are complementary.”
Effective Parallel Corpus Mining using Bilingual Sentence Embeddings (2018, background): “Prior work that explored this approach found that the resulting mined sentence pairs produced poor BLEU scores when used for MT training unless they were combined with traditional human translated corpora with known alignments (Schwenk, 2018).”
The Curious Case of Hallucinations in Neural Machine Translation (2021, unknown): “Design of Noise patterns While the above noise patterns are quite plausible in a web-based corpus collection process, due to the widespread adoption of automatic bitext mining algorithms (Schwenk, 2018) applied over noisy sources, our primary motivation behind constructing these four types of noise ”
Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora (2018, methodology): “Recently, Hassan et al. (2018) and Schwenk (2018) used cosine similarities of sentence embeddings in a common multi-lingual space to select translation pairs for neural machine translation.”
Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options (2020, background): “, 2019), a collection of 135 million Wikipedia parallel sentences, which was constructed by aligning similar sentences in different languages in a joint sentence embedding space (Schwenk, 2018; Artetxe and Schwenk, 2019).”
Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining (2020, background): “An effective approach to parallel data mining is based on multilingual sentence embeddings (Schwenk, 2018; Artetxe and Schwenk, 2019b).”
Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax (2019, methodology): “The second is from [Artetxe and Schwenk, 2018], where they train the multilingual embeddings from an encoder-decoder translation model and get rid of the decoder part after training.”
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, methodology): “More recent work use multilingual sentence embeddings to perform bitext mining, calculating cosine similarity (Schwenk, 2018) or other margin-based similarity (Artetxe”