A Massive Collection of Cross-Lingual Web-Document Pairs

By Ahmed El-Kishky, Vishrav Chaudhary, Francisco (Paco) Guzmán, Philipp Koehn. 2019

230 citations recorded by the source corpus.

Cross-lingual document alignment aims to identify pairs of documents in two distinct languages that are of comparable content or translations of each other. Small-scale efforts have been made to collect aligned document level data on a limited set of language-pairs such as English-German or on limited comparable collections such as Wikipedia. In this paper, we mine twelve snapshots of the Common Crawl corpus and identify web document pairs that are translations of each other. We release a new web dataset consisting of 54 million URL pairs from Common Crawl covering documents in 92 languages paired with English. We evaluate the quality of the dataset by measuring the quality of machine translations from models that have been trained on mined parallel sentence pairs from this aligned corpora and introduce a simple yet effective baseline for identifying these aligned documents. The objective of this dataset and paper is to foster new research in cross-lingual NLP across a variety of low, mid, and high-resource languages.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “Caswell et al. (2021) recruited 51 volunteers speaking 70 languages to judge whether ﬁve publicly available multilingual web-crawled corpora (El-Kishky et al., 2020; Xue et al., 2021; Ortiz Suárez et al., 2020; Bañón et al., 2020; Schwenk et al., 2019) contain text in languages they report, as well…”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “This local mining follows a hierarchical approach: first, document-level language identification along with various rules is applied to find whole documents that are likely to contain mutual translations (El-Kishky et al., 2020).”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, background): “CCAligned (El-Kishky et al., 2020) is a 119language1 parallel dataset built off 68 snapshots”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, neutral): “This is similar to the ccAligned project (El-Kishky et al., 2020).”
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2021, background): “, 2019), ccAligned (El-Kishky et al., 2020), and ParaCrawl (Esplà et al.”
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, methodology): “We use parallel corpora for the translation replaced token detection task, including translation pairs in 100 languages collected from MultiUN (Ziemski et al., 2016), IIT Bombay (Kunchukuttan et al., 2018), OPUS (Tiede-mann, 2012), WikiMatrix (Schwenk et al., 2019), and CCAligned (El-Kishky et al., ”
esCorpius: A Massive Spanish Crawling Corpus (2022, background): “For instance, the CCaligned corpus contains web-document pairs in 137 languages obtained by identifying URLs that are translations of the same page [11].”
Findings of the 2021 Conference on Machine Translation (WMT21) (2021, methodology): “…corpus extracted from the website of Ayatollah Khamenei,5 now only accessible using the 5https://english.khamenei.ir/ 5 6 7 8 9 Wayback Machine.6 For the two FLORES-101 language pairs (i.e. Bengali↔Hindi and Xhosa↔Zulu) all training data is from the CC-Aligned corpus (El-Kishky et al., 2020).”
Ungoliant: An Optimized Pipeline for the Generation of a Very Large-Scale Multilingual Web Corpus (2021, unknown): “, 2020), CCAligned (El-Kishky et al., 2020) and WikiMatrix (Schwenk et al.”
Facebook AI’s WMT20 News Translation Task Submission (2020, methodology): “We use mined bitext data from CCMATRIX (Schwenk et al., 2019) and CCALIGNED (El-Kishky et al., 2019), two complementary mining strategies.”
DeepNet: Scaling Transformers to 1,000 Layers (2022, methodology): “We also expand the data from CCAligned[38],OPUS[33],andTatoeba 3 tocoveralllanguages of Flores101 evaluation sets.”
The Multilingual TEDx Corpus for Speech Recognition and Translation (2021, background): “5B sentences [39, 40] for 100 languages, to directly generate translations for our evaluation sets, we see further improvements.”
Documenting the English Colossal Clean Crawled Corpus (2021, background): “Caswell et al. (2021) recruited 51 volunteers speaking 70 languages to judge whether ﬁve publicly available multilingual web-crawled corpora (El-Kishky et al., 2020; Xue et al., 2020; Ortiz Su´arez et al., 2020; Ba˜n´on et al., 2020; Schwenk et al., 2019) contain text in languages they report, as…”
TorchScale: Transformers at Scale (2022, methodology): “, 2021], CCAligned [El-Kishky et al., 2020], OPUS [Zhang et al.”
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus (2022, background): “, 2020), the common approach to collect large amounts of raw textual data still relies primarily on crawled web text (Ortiz Suárez et al., 2019; Ortiz Suárez et al., 2020; Xue et al., 2021; El-Kishky et al., 2020; Esplà et al., 2019; Bañón et al., 2020; Gao et al., 2020), and although some of the in”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “CC-Aligned [108] Parallel corpus of 292 million non-English common crawl document pairs and 100 million English common crawl document pairs.”