ParaCrawl: Web-Scale Acquisition of Parallel Corpora

By Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu D. Hoang, Miquel Esplà-GomisF, Mikel ForcadaF, Amir Kamran, Faheem Kirefu, Philipp Koehn and 9 more. 2020

296 citations recorded by the source corpus.

We report on methods to create the largest publicly available parallel corpora by crawling the web, using open source software. We empirically compare alternative methods and publish benchmark data sets for sentence alignment and sentence pair filtering. We also describe the parallel corpora released and evaluate their quality and their usefulness to create machine translation systems.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Finetuned Language Models Are Zero-Shot Learners (2021, background): “En–Es from Paracrawl (Bañón et al., 2020) • Summarization asks models to read a piece of text and generate an abbreviated summary of it.”
Documenting the English Colossal Clean Crawled Corpus (2021, background): “…recruited 51 volunteers speaking 70 languages to judge whether ﬁve publicly available multilingual web-crawled corpora (El-Kishky et al., 2020; Xue et al., 2020; Ortiz Su´arez et al., 2020; Ba˜n´on et al., 2020; Schwenk et al., 2019) contain text in languages they report, as well as their quality.”
esCorpius: A Massive Spanish Crawling Corpus (2022, unknown): “01 [14] mC4 [8] CC-100 [15] ParaCrawl v9 [16] esCorpius (ours)”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “Large-scale web-mining projects such as ParaCrawl [14] must also be commended—they have automatically mined a large amount of parallel data from the web for multiple language pairs, including LRLs such as Irish and Nepalese.”
Asymptotically perfect seeded graph matching without edge correlation (and applications to inference) (2025, unknown): “Moreover, it has practical application to problems of harvesting NLP data, such as scraping and aligning parallel sentence pairs from web documents which contain similar content in different languages [90].”