Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings

By Vishrav Chaudhary, Yuqing Tang, Francisco (Paco) Guzmán, Holger Schwenk, Philipp Koehn. 2019

84 citations recorded by the source corpus.

In this paper, we describe our submission to the WMT19 low-resource parallel corpus filtering shared task. Our main approach is based on the LASER toolkit (Language-Agnostic SEntence Representations), which uses an encoder-decoder architecture trained on a parallel corpus to obtain multilingual sentence representations. We then use the representations directly to score and filter the noisy parallel sentences without additionally training a scoring function. We contrast our approach to other promising methods and show that LASER yields strong results. Finally, we produce an ensemble of different scoring methods and obtain additional gains. Our submission achieved the best overall performance for both the Nepali-English and Sinhala-English 1M tasks by a margin of 1.3 and 1.4 BLEU respectively, as compared to the second best systems. Moreover, our experiments show that this technique is promising for low and even no-resource scenarios.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, background): “Filtering (Chaudhary et al., 2019) is performed to remove unaligned data that exists because the original webpage did not have any parallel data, only partial parallel data, or other processing failures.”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, background): “There is also a large area of research in using sentence-level representations to mine pseudo-parallel sentence pairs [6, 13, 19, 22, 48, 49, 52].”
Parallel Corpus Filtering via Pre-trained Language Models (2020, background): “Chaudhary et al. (2019) train a multilingual sentence encoder on various English-Foreign Language parallel corpus and prove the zero-shot cross-lingual transfer capability between non-English pairs, such as Japanese and Chinese.”
The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English (2019, methodology): “…a weakly supervised setting by using a baseline system to ﬁlter out Paracrawl data using LASER (Artetxe and Schwenk, 2018) by following the approach similar to Chaudhary et al. (2019), in order to augment the original training set with a possibly larger but noisier set of parallel sentences.”
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia (2019, background): “This approach has also proven to perform best in a low resource scenario (Chaudhary et al., 2019; Koehn et al., 2019).”
ParaCrawl: Web-Scale Acquisition of Parallel Corpora (2020, background): “Vectors may also be constructed over bigrams (Dara and Lin, 2016) or even higher order n-grams 4http://opus.lingfil.uu.se/ (Uszkoreit et al., 2010).”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, background): “This approach has also proven to perform best in a low resource scenario (Chaudhary et al., 2019; Koehn et al., 2019).”
A Massive Collection of Cross-Lingual Web-Document Pairs (2019, methodology): “We use the open-source LASER toolkit (Schwenk, 2018) with the marginbased filtering criterion to mine sentences, as this method has been shown to accurately align sentences for across a variety of low, mid, and high-resource directions (Schwenk et al., 2019; Chaudhary et al., 2019).”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, methodology): “Noise NMT systems are known to be sensitive to noise, including sentence alignment errors (Khayrallah and Koehn, 2018), so we perform filtering with LASER (Schwenk, 2018; Chaudhary et al., 2019).”
Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions (2019, methodology): “Facebook Chaudhary et al. (2019) use an ensemble of methods: matching of cross-lingual sentence embeddings (their best feature), dual cross entropy based on neural translation model scores, and the open source tools Zipporah and Bicleaner.”
Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation (2020, methodology): “…sentence representations obtained from NMT systems or tailored architectures are achieving a new state-of-the-art in parallel sentence extraction and ﬁltering (Espa˜na-Bonet et al., 2017; Gr´egoire and Langlais, 2018; Artetxe and Schwenk, 2019; Hangya and Fraser, 2019; Chaudhary et al., 2019).”