CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web

By Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin. 2019

320 citations recorded by the source corpus.

We show that margin-based bitext mining in a multilingual sentence space can be successfully scaled to operate on monolingual corpora of billions of sentences. We use 32 snapshots of a curated common crawl corpus (Wenzel et al, 2019) totaling 71 billion unique sentences. Using one unified approach for 90 languages, we were able to mine 10.8 billion parallel sentences, out of which only 2.9 billions are aligned with English. We illustrate the capability of our scalable mining system to create high quality training sets from one language to any other by training hundreds of different machine translation models and evaluating them on the many-to-many TED benchmark. Further, we evaluate on competitive translation benchmarks such as WMT and WAT. Using only mined bitext, we set a new state of the art for a single system on the WMT’19 test set for English-German/Russian/Chinese. In particular, our English/German and English/Russian systems outperform the best single ones by over 4 BLEU points and are on par with best WMT’19 systems, which train on the WMT training data and augment it with backtranslation. We also achieve excellent results for distant languages pairs like Russian/Japanese, outperforming the best submission at the 2020 WAT workshop. All of the mined bitext will be freely available.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, methodology): “In this work, we leverage and extend the corpus provided by two of these mining projects: CCMatrix (Schwenk et al., 2019) and CCAligned17 (El-Kishky et al., 2020).”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, methodology): “Unsupervised parallel data mining To mine parallel data without supervised signals, we employ the margin function formulation [2, 1, 30] based on K nearest neighbors (KNN) to score and rank pairs of sentences from any two languages.”
Nearest Neighbor Machine Translation (2020, methodology): “CCM ATRIX : We train our multilingual model on CCMatrix (Schwenk et al., 2019), containing parallel data for 79 languages and 1,546 language pairs.”
Textless Speech-to-Speech Translation on Real Data (2021, neutral): “On the other hand, Duquenne et al. (2021) extend distance-based bitext mining (Schwenk et al., 2021) to the audio domain by ﬁrst learning a joint embedding space for text and audio, where sentences with similar meaning are close, independent of the modality or language.”
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia (2019, methodology): “This approach was recently extended to mine in Common Crawl texts (Schwenk et al., 2019).”
XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation (2020, neutral): “We will add CCMatrix (Schwenk et al., 2019) in the future.”
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2021, methodology): “Similarly, OPUS contains large parallel datasets generated via automatic ﬁltering and alignment methods, such as WikiMatrix (Schwenk et al., 2021), ccMa-trix (Schwenk et al., 2019), ccAligned (El-Kishky et al., 2020), and ParaCrawl (Esplà et al., 2019), which contain noisy translations.”
Text Embeddings by Weakly-Supervised Contrastive Pre-training (2022, background): “Harvesting semi-structured data sources Large-scale high-quality datasets like C4 [48] and CCMatrix [51] are vital for the success of language model pre-training and machine translation.”
MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases (2022, unknown): “LASER provides joint multilingual sentence embeddings in 93 languages that have been successfully applied to the task of bilingual bitext mining (Schwenk et al., 2019).”
DeepNet: Scaling Transformers to 1,000 Layers (2022, methodology): “More data and language directions: To explore the limits of D EEP N ET on multilingual NMT, we then scale up the training data by using CCMatrix [37].”
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages (2021, background): “4We have not included CCMatrix(Schwenk et al., 2020) and CCAligned(El-Kishky et al.”
Facebook AI’s WMT20 News Translation Task Submission (2020, methodology): “MATRIX (Schwenk et al., 2019) and CCALIGNED (El-Kishky et al.”
Language-agnostic BERT Sentence Embedding (2020, methodology): “, 2018), with performance of models trained on our mined training data being comparable with models trained using CCMatrix (Schwenk et al., 2019).”
Building Machine Translation Systems for the Next Thousand Languages (2022, background): “These improvements have largely been driven by advances in machine learning and the availability of large-scale web-mined datasets (Resnik & Smith, 2003; Uszkoreit et al., 2010; Esplà-Gomis, 2009; Esplà et al., 2019; Bañón et al., 2020; Schwenk et al., 2021).”
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (2024, unknown): “To learn the unified embedding space for cross-lingual semantic matching, the parallel sentences are introduced from two translation datasets, NLLB (NLLB Team et al., 2022) and CCMatrix (Schwenk et al., 2021).”
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, methodology): “We followed the teacher-student approach introduced in (Duquenne et al., 2021) and trained speech encoders with the supervision of the multilingual LASER text encoder (Schwenk et al., 2021b).”
Neural Machine Translation for Low-resource Languages: A Survey (2021, methodology): “This toolkit was the base for subsequent massive-scale parallel corpus extraction projects such as paracrawl [14], wikimatrix [162] and ccmatrix[163].”
We Need To Talk About Random Splits (2020, methodology): “…that only considers the target sentence, but performs better than the best shared task systems: we train an MLP over a LASER sentence embedding (Schwenk et al., 2019) with the following hyperparameters: two hidden layers with 100 parameters each and ReLU activation functions, trained using the…”
Multimodal and Multilingual Embeddings for Large-Scale Speech Mining (2021, methodology): “This bitext mining has provided huge amounts of parallel data, in particular the ParaCrawl project (Esplà et al., 2019) or the CCMatrix initiative (Schwenk et al., 2021).”