OBELISC: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents

By Hugo Laurenccon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord and 1 more. Neural Information Processing Systems, 2023

370 citations recorded by the source corpus.

Large multimodal models trained on natural documents, which interleave images and text, outperform models trained on image-text pairs on various multimodal benchmarks. However, the datasets used to train these models have not been released, and the collection process has not been fully specified. We introduce the OBELICS dataset, an open web-scale filtered dataset of interleaved image-text documents comprising 141 million web pages extracted from Common Crawl, 353 million associated images, and 115 billion text tokens. We describe the dataset creation process, present comprehensive filtering rules, and provide an analysis of the dataset's content. To show the viability of OBELICS, we train vision and language models of 9 and 80 billion parameters named IDEFICS, and obtain competitive performance on different multimodal benchmarks. We release our dataset, models and code.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/948e8cfae92c2004f2dd5c9316f5972f8baaea21

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Aligning Large Multimodal Models with Factually Augmented RLHF (2023, background): “This differential can possibly be attributed to the overlapping “Yes/No” format in VQA and the multiple-choice structure of A-OKVQA.”
SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics (2025, neutral): “This design choice also differs from standard VLM architectures, where each decoder block typically includes both SA and CA layers (Laurençon et al., 2023; Alayrac et al., 2022; Chen et al., 2022).”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, methodology): “To help train the image-history modality, we also use the interleaved image/text data from OBELICS [104].”
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024, methodology): “5-Vision model undergoes a pre-training phase using a diverse dataset, which consists of a combination of interleaved image-text documents ( e.g. , [LST + 24]), image-text pairs from FLD-5B [XWX + 24], synthetic data derived from Optical Character Recognition (OCR) of PDF files, datasets for…”
Kimi k1.5: Scaling Reinforcement Learning with LLMs (2025, positive): “Our multi-modal corpus considered open-sourced interleave datasets like (Zhu et al. 2024; Laurençon et al. 2024) and also constructed large-scale in-house data using resources like textbooks, webpages and tutorials.”
OtterHD: A High-Resolution Multi-modality Model (2023, background): “5 [30] and Idefics-Instruct [24] to achieve better text formatting control.”
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (2024, methodology): “…that our study does not currently capture; as a notable example, we do not study architectures that learn to downsample image patches, such as the Perceiver-based architectures used by Flamingo and IDEFICS (Alayrac et al., 2022; Laurenc¸on et al., 2023) for interleaved image-text training.”
Chameleon: Mixed-Modal Early-Fusion Foundation Models (2024, methodology): “On tasks such as image captioning and visual question answering, Chameleon-34B outperforms models such as Flamingo and IDEFICS, while maintaining competitive performance on text-only benchmarks.”
MMBench: Is Your Multi-modal Model an All-around Player? (2023, neutral): “…(a) Text-Only GPT-4 [37]; (b) Open-Source VLMs including model variants of OpenFlamingo [4], MiniGPT4 [56], InstructBLIP [12], LLaVA [32], IDEFICS [26], CogVLM [47], Qwen-VL [6], Yi-VL [2], mPLUG-Owl [50], InternLM-XComposer [13], and MiniCPM-V [39]; (c) Proprietary VLMs including…”
Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023, neutral): “…of instruction tuning in empowering LLMs, as exempliﬁed by the boosting of GPT-3 [6] to InstrctGPT [22] and ChatGPT [20], which follows natural language instructions effectively to accomplish real-world tasks and allows for customizing task-speciﬁc rules into instructions during downstream…”
Kimi-VL Technical Report (2025, positive): “Our multimodal corpus considered open-sourced interleave datasets like (Zhu et al. 2024; Laurençon et al. 2024) and also constructed large-scale in-house data using resources like textbooks, webpages, and tutorials.”
What matters when building vision-language models? (2024, positive): “Interleaved image-text documents We use OBELICS (Laurençon et al., 2023), an open web-scale dataset of interleaved image-text documents with 350 million images and 115 billion text tokens.”
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (2023, background): “These large datasets typically collect images from the web and parse noisy annotations from the corresponding metadata, such as category label from query [75, 96], short description from alt-text [29,64], as well as detailed description from interleaved text [2, 41].”
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (2024, methodology): “These systems then use an adapter, such as MLPs [30, 31], Q-Former [8, 26], and gated attention [2, 25], to integrate the pretrained CLIP vision encoder into LLMs.”
OpenVLA: An Open-Source Vision-Language-Action Model (2024, background): “While early work explored various architectures for cross-attending between vision and language features [37–41], new open-source VLMs [20, 42–44] have converged on a simpler “patch-as-token” approach, in which patch features from pretrained visual transformers are treated as tokens, and are then…”
Qwen3-VL Technical Report (2025, positive): “We collect diverse real-world multimodal documents sourced from recent Chinese and English websites (Laurençon et al., 2023; Zhu et al., 2023; Li et al., 2024c).”