Towards a Cleaner Document-Oriented Multilingual Crawled Corpus

By Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, Benoît Sagot. 2022

206 citations recorded by the source corpus.

The need for large corpora raw corpora has dramatically increased in recent years with the introduction of transfer learning and semi-supervised learning methods to Natural Language Processing. And while there have been some recent attempts to manually curate the amount of data necessary to train large language models, the main way to obtain this data is still through automatic web crawling. In this paper we take the existing multilingual web corpus OSCAR and its pipeline Ungoliant that extracts and classifies data from Common Crawl at the line level, and propose a set of improvements and automatic annotations in order to produce a new document-oriented version of OSCAR that could prove more suitable to pre-train large generative language models as well as hopefully other applications in Natural Language Processing and Digital Humanities.

Subfields: Artificial Intelligence, Information Systems, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Large Language Models

In Natural Language Processing

Citations in the corpus

BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model (2022, background): “The minimal goal of "training a multilingual large language model" could have been achieved with signiﬁcantly fewer participants; some of the modeling working groups, the engineering working group, and the work needed to ﬁlter an existing data source such as the OSCAR corpus (Abadji et al., 2022).”
esCorpius: A Massive Spanish Crawling Corpus (2022, unknown): “OSCAR 22.01 [14] mC4 [8 4.”
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (2024, methodology): “OSCAR [25] processes Common Crawl using a pipeline inspired by that of Touvron et al. [2], which uses a fastText-based language classifier [26] to filter pages based on their language and then performs deduplication at the line level using a non-cryptographic hash algorithm.”
Building Machine Translation Systems for the Next Thousand Languages (2022, background): “…(Esplà et al., 2019; Bañón et al., 2020), WikiMatrix (Schwenk et al., 2019), CCNET (Wenzek et al., 2020) and CCAligned (El-Kishky et al., 2020), OSCAR (Ortiz Suárez et al., 2019; Ortiz Suárez et al., 2020; Abadji et al., 2022), and several others; all of which have between 100 and 300 languages.”
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants (2023, background): “for mid- to low-resource languages as capable as in English will therefore require scaling corpus sizes in more languages when feasible (Abadji et al., 2022) or”
Weight Tying Biases Token Embeddings Towards the Output Space (2026, unknown): “Here we calculate the norm-distribution of the embedding vectors as a function of the frequency of occurrence of tokens in a 300MB English-text sample from OSCAR-2301 (Abadji et al., 2022).”