The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

By Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf. Neural Information Processing Systems, 2024

1,023 citations recorded by the source corpus.

The performance of a large language model (LLM) depends heavily on the quality and size of its pretraining dataset. However, the pretraining datasets for state-of-the-art open LLMs like Llama 3 and Mixtral are not publicly available and very little is known about how they were created. In this work, we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl snapshots that produces better-performing LLMs than other open pretraining datasets. To advance the understanding of how best to curate high-quality pretraining datasets, we carefully document and ablate all of the design choices used in FineWeb, including in-depth investigations of deduplication and filtering strategies. In addition, we introduce FineWeb-Edu, a 1.3-trillion token collection of educational text filtered from FineWeb. LLMs pretrained on FineWeb-Edu exhibit dramatically better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC. Along with our datasets, we publicly release our data curation codebase and all of the models trained during our ablation experiments.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

View Paper: https://www.semanticscholar.org/paper/b83a9e35c3aeeb37708e362473c7617d59b815b5

Citations in the corpus

• Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, neutral): “All models use the Llama tokenizer and are trained on the same subset of the FineWeb-Edu dataset (Penedo et al., 2024).”
• Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, neutral): “Encoder models offer a compelling trade-off in quality versus size, making them a popular https://github.com/AnswerDotAI/ModernBERT option against encoder-decoder and decoder-only language models when dealing with substantial amounts of data (Penedo et al., 2024).”
• WebKnoGraph: GNN-Powered Internal Linking (2026, unknown): “To place the Kalicube site graph into a broader web-scale context, we draw on the FineWeb dataset [30], a large-scale web corpus designed for research use and derived from multiple Common Crawl snapshots.”
• Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies (2026, unknown): “Specifically, we sample passages from FineWeb-Edu (Penedo et al., 2024).”
• Avey-B (2026, unknown): “We pretrain two Avey-B sizes, base (165M) and large (391M), for 180B tokens drawn from the FineWeb 300BT split (Penedo et al., 2024), using PyTorch DDP across 16 NVIDIA H200 GPUs (Paszke et al., 2019; Li et al., 2020).”
• Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Report (2025, unknown): “DCLM-Pro [40] and FineWeb [22] are full documents randomly from their respective datasets, while dclm_sent is comprised of individual DCLM-Pro documents broken down into individual sentences, to create more varied small-length inputs, again following Stella [38]. ettin_tokens and english_words were…”
• mmBERT: A Modern Multilingual Encoder with Annealed Language Learning (2025, unknown): “However, data quality has significantly increased since 2019, allowing us to achieve higher scores with only half of the tokens (Penedo et al., 2024).”
• Seq vs Seq: An Open Suite of Paired Encoders and Decoders (2025, unknown): “In the process of training, the Olmo 2 paper (OLMo et al., 2025) described their approach of using filtered DCLM and other higher-quality sources for the decay phase (similar to FineWeb-Edu filtering Penedo et al. (2024); Lozhkov et al. (2024)).”
• ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance (2025, unknown): “Semantic filtering was done by fine-tuning a BERT classifier trained on a document quality dataset automatically labeled by LLama-3 70B (Grattafiori et al., 2024).”
• DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search (2026, unknown): “For FineWeb-Edu, which is already curated and comprises over half of the mixture, we skip rule-based filtering and apply a more aggressive percentile threshold.”
• AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages (2026, unknown): “In multilingual settings, FineWeb2 (Penedo et al., 2025) extends these pipelines to scale pre-training data processing to over 1,000 languages.”