LAION-5B: An open large-scale dataset for training next generation image-text models
By Christoph Schuhmann, Romain Beaumont, R. Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman and 6 more. 2022
5,389 citations recorded by the source corpus.
Groundbreaking language-vision architectures like CLIP and DALL-E proved the utility of training on large amounts of noisy image-text data, without relying on expensive accurate labels used in standard vision unimodal supervised learning. The resulting models showed capabilities of strong text-guided image generation and transfer to downstream tasks, while performing remarkably at zero-shot classification with noteworthy out-of-distribution robustness. Since then, large-scale language-vision models like ALIGN, BASIC, GLIDE, Flamingo and Imagen made further improvements. Studying the training and capabilities of such models requires datasets containing billions of image-text pairs. Until now, no datasets of this size have been made openly available for the broader research community. To address this problem and democratize research on large-scale multi-modal models, we present LAION-5B - a dataset consisting of 5.85 billion CLIP-filtered image-text pairs, of which 2.32B contain English language. We show successful replication and fine-tuning of foundational models like CLIP, GLIDE and Stable Diffusion using the dataset, and discuss further experiments enabled with an openly available dataset of this scale. Additionally we provide several nearest neighbor indices, an improved web-interface for dataset exploration and subset generation, and detection scores for watermark, NSFW, and toxic content detection. Announcement page https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Yoshua Bengio
- Quoc V. Le
- Sergey Levine
- Percy Liang
- Luke S. Zettlemoyer
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In Multimodal Learning
In Evaluation & Benchmarks
- Jun-Yan Zhu
- Richard Zhang
- Alexei A. Efros
- Ming-Hsuan Yang
- Luke S. Zettlemoyer
- Phillip Isola
- Antonio Torralba
- Mohit Bansal
Citations in the corpus
- Visual Instruction Tuning (2023, background): “The community has witnessed a surge in the amount of public multimodal data such as image-text pairs, ranging from CC [8] to LAION [45].”
- Stable and low-precision training for large-scale vision-language models (2023, methodology): “To evaluate SwitchBack we train CLIP [46] visual transformer [20] models on LAION-2B [53].”
- Language Is Not All You Need: Aligning Perception with Language Models (2023, methodology): “The image-caption pairs are constructed from several datasets, including English LAION-2B [19], LAION-400M [20], COYO-700M [21], and Conceptual Captions [22, 23].”
- InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “5+ [56] dataset and manually wrote instructions and output captions.”
- OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models (2023, methodology): “The cross-modal module is trained on web-scraped image-text sequences, in our case, two open source datasets: LAION-2B [32] and Multimodal C4 [45].”
- Learning Interactive Real-World Simulators (2023, background): “For instance, paired text-image data from the internet contains rich scenes and objects but little movement [4, 5], video captioning and question answering data contain rich high-level descriptions but little low-level movement detail [6, 7], human activity data contains rich human action but…”
- Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, methodology): “During training, only the public vision-language data [9,13,68,70] is required for fine-tuning LLaMA to learn the 3D-conditioned response capacity.”
- IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models (2023, methodology): “To train the IP-Adapter, we build a multimodal dataset including about 10 million text-image pairs from two open source datasets - LAION-2B [42] and COYO-700M [43].”
- Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “For VLM pre-training, multiple large-scale image-text datasets [10], [17], [20], [21] were collected from the internet.”
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, methodology): “We hypothesize that this is due to CLIP’s training data , such as LAION [107], containing abundant OCR and text-heavy data, whereas SSL and other vision models primarily train on nat-ural images with significantly less text content.”
- Kosmos-2: Grounding Multimodal Large Language Models to the World (2023, methodology): “To learn the grounding capability, we first construct a large-scale dataset of Gr ounded I mage-T ext pairs (G R IT), based on image-text pairs from a subset of COYO-700M (Byeon et al., 2022) and LAION-2B (Schuhmann et al., 2022).”
- A survey on multimodal large language models (2023, positive): “Furthermore, some methods use a projection-based interface to close the modality gap [21,30,38,43].”
- Human Preference Score: Better Aligning Text-to-image Models with Human Preference (2023, methodology): “The FID [13] is computed on 10k images from the LAION [38] dataset.”
- CogVLM: Visual Expert for Pretrained Language Models (2023, neutral): “Conventionally, LLMs are pretrained on extensive text-only datasets [Raffel et al., 2020], leading to a significant divergence in data distribution when compared to image-text pair datasets like LAION [Schuhmann et al., 2022] and COYO [Byeon et al., 2022].”
- CogAgent: A Visual Language Model for GUI Agents (2023, positive): “This is similar to the Synthetic Document Generator in Kim et al. [15], with text of varying font, size, color and orientation, and diverse image background from LAION-2B [32].”