Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures

By Pedro Ortiz Suarez, Benoît Sagot, Laurent Romary. 2019

493 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “In order to select high quality documents from these extractions, we trained a 2-gram fastText [48] classiﬁer.”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “Fertility is defined as the number of subwords created per word or per dataset by the tokenizer, which we measured using subsets of Universal Dependencies 2.9 (Nivre et al., 2017) and OSCAR (Ortiz Suárez et al., 2019) in the languages of interest.”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “We first consider popular public web datasets (OSCAR-2019 (Ortiz Su ´ arez et al., 2019), OSCAR-2022 (Abadji et al., 2021), C4 (Raffel et al., 2020)), The Pile (Gao et al., 2020) as the most popular publicly available curated dataset, and variations of RefinedWeb (RW-Raw, RW-Filtered, and RW as…”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “, 2021) and OSCAR (Suárez et al., 2019); and third, it has been widely used by state-of-the-art models including GPT-J-6B (Wang & Komatsuzaki, 2021), GPT-NeoX-20B (Black et al.”
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, background): “Modern pretraining corpora are typically massive preprocessed generalist webcrawls [Ortiz Suárez et al., 2019, Raffel et al., 2020], collected with no explicit regard for downstream tasks–although adding curated high-quality cross-domain data has been proposed as a path towards better zero-shot…”
The ROOTS Search Tool: Data Transparency for LLMs (2023, methodology): “During preliminary experiments on the ROOTS corpus, OSCAR (Ortiz Suárez et al., 2019) has been identiﬁed as a source of a large amount of documents containing personally iden-tiﬁable information (PII).”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, methodology): “…a spectrum from no details at all (Brown et al., 2020) to detailed filtering instructions, with (Raffel et al., 2020) or without the dataset release (Rae et al., 2021) to detailed filtering instructions with the accompanying code (Gao et al., 2020; Conneau et al., 2020; Ortiz Suárez et al., 2019).”
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus (2021, methodology): “…et al., 2018; Jo and Gebru, 2020; Paullada et al., 2020; Bender et al., 2021), establish standardized ﬁltering pipelines for corpora (Roziewski and Stokowiec, 2016; Ortiz Suarez et al., 2019; Wenzek et al., 2020), and develop methods for evaluating the bias in trained models (Schick et al., 2021).”
What Language Model to Train if You Have One Million GPU Hours? (2022, methodology): “We evaluate three possible corpora, all commonly used to train large language models: • OSCAR v1 (Ortiz Suárez et al., 2019) 2 , a multilingual, ﬁltered version of Common Crawl; • C4 (Raffel et al., 2019), speciﬁcally its replication by AllenAI, a processed and ﬁltered version of Common Crawl; •…”
Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus (2020, methodology): “This section brieﬂy analyzes the OSCAR corpus (Ortiz Su´arez et al., 2019), which, although an excellent resource for many languages, has lower-quality content for some languages.”
Ungoliant: An Optimized Pipeline for the Generation of a Very Large-Scale Multilingual Web Corpus (2021, unknown): “…for other languages, namely those of Grave et al. (2018), CCNet (Wenzek et al., 2020), Multilingual C4 (mC4) (Xue et al., 2020) and OSCAR (Ortiz Su´arez et al., 2019, 2020) for pre-training language models, as well as, Paracrawl (Espl`a et al., 2019; Ba˜n´on et al., 2020), CCAligned (El-Kishky…”
Building Machine Translation Systems for the Next Thousand Languages (2022, background): “…(Esplà et al., 2019; Bañón et al., 2020), WikiMatrix (Schwenk et al., 2019), CCNET (Wenzek et al., 2020) and CCAligned (El-Kishky et al., 2020), OSCAR (Ortiz Suárez et al., 2019; Ortiz Suárez et al., 2020; Abadji et al., 2022), and several others; all of which have between 100 and 300 languages.”