Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets

By Isaac Caswell, Julia Kreutzer, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Auguste Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote and 42 more. 2021

356 citations recorded by the source corpus.

With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, Web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50% sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous language codes. We demonstrate that these issues are easy to detect even for non-proficient speakers, and supplement the human audit with automatic analyses. Finally, we recommend techniques to evaluate and improve multilingual corpora and discuss potential risks that come with low-quality data releases.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, background): “Furthermore, zero-shot generalization is the “natural” setting for autoregressive decoder-only models, in which they perform best (Wang et al., 2022).”
Multimodal datasets: misogyny, pornography, and malignant stereotypes (2021, methodology): “With regards to CCAligned, a 119-language parallel dataset built off 68 snapshots of Common Crawl, Caswell et al. [40] revealed that there were notable amounts of pornographic content (> 10”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, methodology): “Crawled content also over-represents pornographic text across languages (Kreutzer et al., 2022), especially in the form of spam ads.”
Datasets: A Community Library for Natural Language Processing (2021, background): “Recent work has outlined the difﬁculty of ﬁnding data sources for lower-resourced languages through automatic ﬁltering alone (Caswell et al., 2021).”
PaLM 2 Technical Report (2023, background): “Memorization of the tail Though multilinguality presents many benefits to PaLM 2, it also presents a new risk for memorization: data is often scarcer and of poorer quality (Kreutzer et al., 2022).”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “There has also been an increased focus on analyzing other dimensions of data quality and stewardship [102, 107, 121, 123], with several noteworthy initiatives aiming to document both existing [9, 20, 30, 43], and newly developed [16, 60, 136] resources.”
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus (2021, background): “While certain notable endeavors have indeed analyzed speciﬁc aspects of corpora such as the Common Crawl (Kolias et al., 2014; Caswell et al., 2021) and Wikipedia (Hube, 2017), they have only scratched the surface of what these bodies of text contain.”
Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources (2022, background): “…Gehrmann et al., 2021; Wang et al., 2021) and even to retrospectively document and analyze datasets that were developed and released by others without thorough documentation (Bandy and Vincent, 2021; Kreutzer et al., 2021; Birhane et al., 2021; Dodge et al., 2021).”
In-context Examples Selection for Machine Translation (2022, background): “Table 8 shows that the source lexical overlap is a good indicator of the informativeness of a prompt for 3 out of 4 domains, with Koran as an exception.”
Building Machine Translation Systems for the Next Thousand Languages (2022, methodology): “A recent audit conducted by Kreutzer et al. (2022) on five public, multilingual datasets found pervasive issues.”
Biases in Large Language Models: Origins, Inventory, and Discussion (2023, background): “Although modern language models are trained on massive corpora [45, 78, 82, 107, 108], the documents that make up their training dataset are still a subset of the text available on the Web [27, 51, 132].”