The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

By Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra-Aimée Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, Julien Launay. 2023

967 citations recorded by the source corpus.

Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclear how scalable is curation and whether we will run out of unique high-quality data soon. At variance with previous beliefs, we show that properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models from the state-of-the-art trained on The Pile. Despite extensive filtering, the high-quality data we extract from the web is still plentiful, and we are able to obtain five trillion tokens from CommonCrawl. We publicly release an extract of 600 billion tokens from our RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “Open-source releases like BLOOM (Scao et al., 2022) and Falcon (Penedo et al., 2023) have risen to challenge their closed-source counterparts like GPT-3 and Chinchilla.”
Code Llama: Open Foundation Models for Code (2023, background): “It is well-known that data quality is critical in the training and responsible development of LLMs (e.g., Hoffmann et al., 2022; Penedo et al., 2023), and this is also true for code as discussed by Allal et al.”
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants (2023, background): “However, this only seems to be the case for FALCON.”
∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024, background): “LLMs are typically trained on sequences containing no more than 8K tokens (Tou-vron et al., 2023; Penedo et al., 2023; Biderman et al., 2023), and thus cannot well handle contexts exceeding 8K tokens.”
Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions (2023, methodology): “These datasets often contain model’s responses to malicious questions that are generic without providing reasonable refusal responses.”
Poisoning Web-Scale Training Datasets is Practical (2023, neutral): “[70] 276 2023 + ✗ 0.24% ≥ 0.02% - OBELISC [49] 353 2023 + ✗ 0.09% ≥ 0.01% - LAION-2B-en [83] 2323 2022 [46] 0 .”
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, methodology): “As we full-fine-tune the language model backbone for long steps, we also jointly train with a text-only dataset RefinedWeb (Penedo et al., 2023) to avoid harming its text reasoning capability due to catastrophic forgetting.”
OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text (2023, methodology): “Likewise, Trafilatura (Barbaresi, 2021), which was used in RefinedWeb (Penedo et al., 2023), had poor efficiency.”
Scaling Data-Constrained Language Models (2023, background): “[83] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.”
DeepInception: Hypnotize Large Language Model to Be Jailbreaker (2023, methodology): “Specifically, we employ three open-source 7B parameters LLMs, e.g., Llama-2 [18], Falcon [13] and Vicuna [25], and two closed-source LLMs, e.g., GPT-3.”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, neutral): “RLHF is employed in recent LLM studies, including general LLMs [214], [256] and medical LLMs [44], [186].”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, positive): “• WizardMath surpasses all other open-source LLMs by a substantial margin in terms of mathematical reasoning, including Llama-2 70B [20], Llama-1 65B [4], Falcon-40B [21], MPT30B8, Baichuan-13B Chat9 and ChatGLM2 12B [45] on both GSM8k [42] and MATH [43].”
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges (2024, background): “Falcon : Falcon, equipped with 40 billion parameters, undergoes pre-training on a large dataset of 1.3 trillion tokens [77].”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “The progress in the field has been further accelerated by the emergence of open-source LLMs, including the LLaMA series [106, 107], Vicuna [145], InternLM [104], MOSS [101], ChatGLM [36], Qwen [4], Baichuan [6], and Falcon [86], among others [26, 103, 119].”
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism (2024, background): “We have gained valuable insights from reputable sources such as (Computer, 2023; Gao et al., 2020; Penedo et al., 2023; Touvron et al., 2023a).”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, neutral): “pre-training corpora are automatically collected from the web and often contain a significant amount of fabricated, outdated, or biased information (Penedo et al., 2023).”
Baichuan 2: Open Large-scale Language Models (2023, background): “, 2022), MPT (MosaicML, 2023) and Falcon (Penedo et al., 2023), enables researchers to freely access the models for examination, experimentation, and further development.”
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, methodology): “(5)OpenLLaMA v2 is pre-trained with a mixture of RefinedWeb (Penedo et al., 2023), StarCoder (Li et al.”