WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning

By Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, Marc Najork. 2021

441 citations recorded by the source corpus.

The milestone improvements brought about by deep representation learning and pre-training techniques have led to large performance gains across downstream NLP, IR and Vision tasks. Multimodal modeling techniques aim to leverage large high-quality visio-linguistic datasets for learning complementary information across image and text modalities. In this paper, we introduce the Wikipedia-based Image Text (WIT) Dataset to better facilitate multimodal, multilingual learning. WIT is composed of a curated set of 37.5 million entity rich image-text examples with 11.5 million unique images across 108 Wikipedia languages. Its size enables WIT to be used as a pretraining dataset for multimodal models, as we show when applied to downstream tasks such as image-text retrieval. WIT has four main and unique advantages. First, WIT is the largest multimodal dataset by the number of image-text examples by 3x (at the time of writing). Second, WIT is massively multilingual (first of its kind) with coverage over 100+ languages (each of which has at least 12K examples) and provides cross-lingual texts for many images. Third, WIT represents a more diverse set of concepts and real world entities relative to what previous datasets cover. Lastly, WIT provides a very challenging real-world test set, as we empirically illustrate using an image-text retrieval task as an example. WIT Dataset is available for download and use via a Creative Commons license here: https://github.com/google-research-datasets/wit.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “…down-stream generalization capabilities [6, 15, 37, 60], prompted us to construct a large-scale pretraining dataset from various publicly available image-text datasets, including a ﬁltered version of LAION [43], Wikipedia Image-Text [48], CC3M [8], Visual Genome [26] and Localized Narratives [35].”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, methodology): “Besides CC12M, there also exist some other largescale image-text datasets, such as WIT [36], WenLan [16], LAION-400M [34], and the datasets used in CLIP [33] and ALIGN [17].”
LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, methodology): “Second, we use the Wikipedia based Image Text (WIT) dataset [53] to perform T→ I retrieval across more than a hundred languages.”
MURAL: Multimodal, Multitask Retrieval Across Languages (2021, methodology): “The large-scale Wikipedia Image Text (WIT) dataset (Srinivasan et al., 2021) is mined from Wikipedia, covering 108 languages.”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, methodology): “To verify our assumption, we continue the pretraining process on the cleaner WIT dataset (Srinivasan et al., 2021) for 50k steps.”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, background): “To the best of our knowledge, the largest multilingual dataset before LAION-5B has around 36 million samples from Wikipedia Image Text [75].”
Multimodal datasets: misogyny, pornography, and malignant stereotypes (2021, methodology): “…NSFW ﬁeld that not just alludes towards its unreliability as a ﬁltering parameter but also highlights the need for combined text and image based ﬁltering steps used in projects such as the Wikipedia-based Image Text dataset [41] (also abbreviated as WIT) and the Conceptual Captions dataset [25].”
KNN-Diffusion: Image Generation via Large-Scale Retrieval (2022, methodology): “The modified PMD dataset is composed from the following set of publicly available Text-Image datasets: SBU Captions [33], Localized Narratives [35], Conceptual Captions [38], Visual Genome [25], Wikipedia Image Text [42], Conceptual Captions 12M [7], Red Caps [9], and, a filtered version of YFCC100M”
Objaverse: A Universe of Annotated 3D Objects (2022, methodology): “Massive web-scraped datasets containing image-text pairs such as Conceptual Captions [63], WIT [65], and LAION [61,62] have seen increased popularity recently as they have been used to train impressive models for vision-language representation learning [27,30,55], text-to-image generation…”
The Platonic Representation Hypothesis (2024, unknown): “: We measure alignment using mutual nearest-neighbor on the Wikipedia caption dataset (WIT) (Srinivasan et al., 2021).”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “Differently from previous datasets, Wikipedia-based Image Text (WIT) [141] provides images coming from Wikipedia together with various metadata extracted from the original pages, with approximately 5.”
RedCaps: web-curated image-text data created by the people, for the people (2021, background): “Datasets in 2021 # Instances Released RedCaps (ours) 11,679,006 CC-12M [30] 12,423,374 WIT-english [44] 5,500,746 CLIP [28] 400M × ALIGN [29] 1.”
Sigmoid Loss for Language Image Pre-Training (2023, background): “Seminal works CLIP [34] and ALIGN [23] established the viability of this approach at a large scale, and following their success, many large image-text datasets became available privately [55, 13, 21, 45] and publicly [37, 6, 15, 7, 38].”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, background): “To this end, researches usually leverage existing web images with alt-text [12–16] and videos with ASR transcriptions [2, 4, 5, 3, 6, 17, 18] for scalable learning.”
Reveal: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory (2022, methodology): “By decoupling the knowledge memorization from reasoning, we enable our model to leverage various external sources of knowledge (Wikipedia passages and images [36], the WikiData knowledge graph [39], Web image-text pairs [5] and visual question answering data [12]).”
DataComp: In search of the next generation of multimodal datasets (2023, background): “• WIT [127]: Image-text pairs from Wikipedia pages.”
FindIt: Generalized Localization with Natural Language Queries (2022, background): “Large amounts of work have been done to learn across image and language, including visualgrounding [16,81,49,96,91,62], image captioning [6,1,5], visual question and answering (VQA) [33,12], visual dialog [11,39], visual reasoning [75,93,85], text-toimage and image-to-text retrieval [32,66,74], and ”