WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia

By Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, Francisco (Paco) Guzmán. 2019

443 citations recorded by the source corpus.

We present an approach based on multilingual sentence embeddings to automatically extract parallel sentences from the content of Wikipedia articles in 96 languages, including several dialects or low-resource languages. We do not limit the extraction process to alignments with English, but we systematically consider all possible language pairs. In total, we are able to extract 135M parallel sentences for 16720 different language pairs, out of which only 34M are aligned with English. This corpus is freely available. To get an indication on the quality of the extracted bitexts, we train neural MT baseline systems on the mined data only for 1886 languages pairs, and evaluate them on the TED corpus, achieving strong BLEU scores for many language pairs. The WikiMatrix bitexts seem to be particularly interesting to train MT systems between distant languages without the need to pivot through English.

Subfields: Artificial Intelligence, Communication, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “crawled corpora (El-Kishky et al., 2020; Xue et al., 2021; Ortiz Suárez et al., 2020; Bañón et al., 2020; Schwenk et al., 2019) contain text in languages they report, as well as their quality.”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “In this work, we leverage and extend the corpus provided by two of these mining projects: CCMatrix (Schwenk et al., 2019) and CCAligned17 (El-Kishky et al., 2020).”
MLQA: Evaluating Cross-lingual Extractive Question Answering (2019, methodology): “Our approach is similar to WikiMatrix (Schwenk et al., 2019) which extracts parallel sentences for many language pairs in Wikipedia, but we limit the search for parallel sentences to documents on the same topic only, and aim for N -way parallel sentences.”
Parallel Corpus Filtering via Pre-trained Language Models (2020, methodology): “Other works based on sentence embeddings include Hangya and Fraser (2018) and Littell et al. (2018), as well as Schwenk et al. (2019), which mines millions of parallel sentences in 1620 language pairs from Wikipedia.”
MURAL: Multimodal, Multitask Retrieval Across Languages (2021, methodology): “…used in the construction of EOBT includes: Europarl (Koehn, 2005), Paracrawl (Es-plà et al., 2019), TED57, Tanzil (Tiedemann, 2012), NewsCommentary, Wikimatrix (Schwenk et al., 2021), Wikititles, JW300 (Agi´c and Vuli´c, 2019), Opus100 (Zhang et al., 2020), SETimes (Tyers and Alperen, 2010), UNv1.”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, background): “WikiMatrix (Schwenk et al., 2019) CCAligned (ElKishky et al.”
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, methodology): “We collect translation pairs from MultiUN (Ziemski et al., 2016), IIT Bom-bay (Kunchukuttan et al., 2018), OPUS (Tiede-mann, 2012), and WikiMatrix (Schwenk et al., 2019).”
Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation (2020, background): “• WikiMatrix: Mined parallel sentences from Wikipedia in different languages (Schwenk et al., 2019).”
ParaCrawl: Web-Scale Acquisition of Parallel Corpora (2020, methodology): “This lead to work to construct WikiMatrix, a large corpus of parallel sentences from Wikipedia (Schwenk et al., 2019) based on cosine distance of their cross-lingual sentence embeddings.”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, methodology): “Therefore, we increased the value of the neighborhood k in Equation 1 from 4, which was used in (Schwenk et al., 2019), to 16.”
A Massive Collection of Cross-Lingual Web-Document Pairs (2019, methodology): “Our expectation is that, if successful, our approach should be able to mine parallel sentences that are of comparable quality to recent approaches that leverage Wikipedia (Schwenk et al., 2019) and ParaCrawl2 (EsplàGomis et al.”
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2021, methodology): “Similarly, OPUS contains large parallel datasets generated via automatic ﬁltering and alignment methods, such as WikiMatrix (Schwenk et al., 2021), ccMa-trix (Schwenk et al., 2019), ccAligned (El-Kishky et al., 2020), and ParaCrawl (Esplà et al., 2019), which contain noisy translations.”
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, methodology): “We use parallel corpora for the translation replaced token detection task, including translation pairs in 100 languages collected from MultiUN (Ziemski et al., 2016), IIT Bombay (Kunchukuttan et al., 2018), OPUS (Tiede-mann, 2012), WikiMatrix (Schwenk et al., 2019), and CCAligned (El-Kishky et al., ”
Emerging Cross-lingual Structure in Pretrained Language Models (2019, methodology): “, 2018), a cross-lingual natural language inference dataset, for alignment supervision and Tatoeba (Schwenk et al., 2019), a sentence retrieval dataset, for evaluation.”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, methodology): “We used a margin threshold of 1.05 for WikiMatrix and a threshold of 1.04 for the remaining datasets, as we expect them to be cleaner.”
Building Machine Translation Systems for the Next Thousand Languages (2022, background): “, 2020), but their quality is rarely measured directly, and is rather judged through the improvements they bring to downstream applications (Schwenk et al., 2019).”
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora (2020, methodology): “…corpus, we use the same corpus as I NFO XLM (Chi et al., 2020b), including MultiUN (Ziemski et al., 2016), IIT Bombay (Kunchukuttan et al., 2017), OPUS (Tiedemann, 2012), and WikiMatrix (Schwenk et al., 2019) We use a transformer-encoder (Vaswani et al., 2017) as the backbone of the model.”
Findings of the 2020 Conference on Machine Translation (WMT20) (2020, background): “These included Wikimatrix (Schwenk et al., 2019), which was added for all language pairs where it was available.”
Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation (2020, methodology): “For en – fr , the supervised system trained on WikiMatrix outperforms SSNMT trained on WP by 3–4 BLEU points, while the opposite is the case for en – de , where SSNMT achieves 1–5 The difference in the translation quality using only the data selected at different epochs reﬂects the evolving closenes”
SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings (2020, background): “A lack of parallel data cannot be easily remedied: mining parallel sentences is possible (Schwenk et al., 2019) but assumes that monolingual corpora contain parallel sentences.”