Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora

By Marcin Junczys-Dowmunt. 2018

140 citations recorded by the source corpus.

In this work we introduce dual conditional cross-entropy filtering for noisy parallel data. For each sentence pair of the noisy parallel corpus we compute cross-entropy scores according to two inverse translation models trained on clean data. We penalize divergent cross-entropies and weigh the penalty by the cross-entropy average of both models. Sorting or thresholding according to these scores results in better subsets of parallel data. We achieve higher BLEU scores with models trained on parallel data filtered only from Paracrawl than with models trained on clean WMT data. We further evaluate our method in the context of the WMT2018 shared task on parallel corpus filtering and achieve the overall highest ranking scores of the shared task, scoring top in three out of four subtasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, methodology): “Junczys-Dowmunt [79] implements a cross-entropy data filtering method for bilingual data, which uses cross-entropy scores calculated for noisy pairs according to two translationmodels trained on the clean data.”
Parallel Corpus Filtering via Pre-trained Language Models (2020, methodology): “Junczys-Dowmunt (2018) has a dual conditional cross-entropy adequacy ﬁlter and a domain ﬁlter trained on news corpora.”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, background): “And there is a large literature on filtering text data, e.g. (Axelrod et al., 2011; Moore and Lewis, 2010; Wang et al., 2018; Kamholz et al., 2014; Junczys-Dowmunt, 2018; Caswell et al., 2020).”
Findings of the 2019 Conference on Machine Translation (WMT19) (2019, methodology): “filtered backtranslation with restricted sampling (like Edunov+ 2018), filtered ParaCrawl and CommonCrawl (Junczys-Dowmunt, 2018a), continued training on newstest15-18 (like JHU’s submission to WMT18, Koehn et al.”
ParaCrawl: Web-Scale Acquisition of Parallel Corpora (2020, methodology): “…language models on clean data (Rossenbach et al., 2018), language models trained on the provided raw data as contrast, neural translation models (Junczys-Dowmunt, 2018), bag-of-words lexical translation probabilities (Gonz´alez-Rubio, 2019), or even existing off-the-shelf tools like Zipporah…”
Detecting Hallucinated Content in Conditional Neural Sequence Generation (2020, methodology): “This is in contrast to previous data ﬁltering methods for MT, which remove entire sentence pairs (Junczys-Dowmunt, 2018; Kang and Hashimoto, 2020).”
Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings (2019, methodology): “eu/ parallel sentences (Junczys-Dowmunt, 2018) for high-resource languages, their effectiveness has”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, methodology): “4Conditional probabilities of MT systems in each direction have been shown effective at filtering MT training data (Junczys-Dowmunt, 2018).”
Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions (2019, methodology): “While the state-of-the-art methods that use NMT models have proven effective in mining parallel sentences (Junczys-Dowmunt, 2018) for high-resource languages, their effectiveness has not been tested in low-resource languages.”
Parallel Sentence Mining by Constrained Decoding (2020, methodology): “based on this (Junczys-Dowmunt, 2018) won the WMT 2018 shared task on parallel corpus filtering (Koehn et al.”
Unsupervised Domain Clusters in Pretrained Language Models (2020, methodology): “, 2019b), applying them on noisy, web-crawled data (Junczys-Dowmunt, 2018) or for additional tasks (Gururangan et al.”
Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation (2019, methodology): “We try to combine the strengths of last year’s two dominating systems for the English-German news translation task – FAIR’s submission with large-scale noisy back-translation (Edunov et al., 2018) and our own, based on dual cross-entropy data-ﬁltering (Junczys-Dowmunt, 2018b,a).”
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages (2021, methodology): “As future work, we plan to study different data filtering (Junczys-Dowmunt, 2018) and data sampling techniques (Bengio et al.”
AutoQA: From Databases to Q&A Semantic Parsers with Only Synthetic Training Data (2020, methodology): “We use a subset of 5 million sentence pairs with the highest dual conditional cross-entropy score (Junczys-Dowmunt, 2018), and use only one of the five paraphrases provided for each sentence.”
Microsoft’s Submission to the WMT2018 News Translation Task: How I Learned to Stop Worrying and Love the Data (2018, methodology): “In Junczys-Dowmunt et al. (2018), we prepared a baseline setup for Marian which reproduces the highest scoring NMT system (Sennrich et al.”
The Curious Case of Hallucinations in Neural Machine Translation (2021, unknown): “Robust Learning on Noisy Samples Kang and Hashimoto (2020) propose a loss-truncation approach to reduce the impact of noisy references in sequence-to-sequence training, using the intermediate model’s loss as a sample quality estimator and test their algorithm on a summarization task.”
Large-Scale, Diverse, Paraphrastic Bitexts via Sampling and Clustering (2019, methodology): “Based on experience with data quality issues in neural MT (Ott et al., 2018; Junczys-Dowmunt, 2018), we decided to further clean the corpus.”