The Open Images Dataset V4

By Alina Kuznetsova, Hassan Rom, Neil Gordon Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov and 2 more. 2018

1,627 citations recorded by the source corpus.

We present Open Images V4, a dataset of 9.2M images with unified annotations for image classification, object detection and visual relationship detection. The images have a Creative Commons Attribution license that allows to share and adapt the material, and they have been collected from Flickr without a predefined list of class names or tags, leading to natural class statistics and avoiding an initial design bias. Open Images V4 offers large scale across several dimensions: 30.1M image-level labels for 19.8k concepts, 15.4M bounding boxes for 600 object classes, and 375k visual relationship annotations involving 57 classes. For object detection in particular, we provide 15×\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$15\times $$\end{document} more bounding boxes than the next largest datasets (15.4M boxes on 1.9M images). The images often show complex scenes with several objects (8 annotated objects per image on average). We annotated visual relationships between them, which support visual relationship detection, an emerging task that requires structured reasoning. We provide in-depth comprehensive statistics about the dataset, we validate the quality of the annotations, we study how the performance of several modern models evolves with increasing amounts of training data, and we demonstrate two applications made possible by having unified annotations of multiple types coexisting in the same images. We hope that the scale, quality, and variety of Open Images V4 will foster further research and innovation even beyond the areas of image classification, object detection, and visual relationship detection.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “We generate this dataset by combining foreground objects extracted from OpenImages V5 [3] with backgrounds, licensed for reuse with modification, mined from search engine results.”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, background): “Novel Object Captioning (NoCaps) [1] extends the image captioning task, and provides a benchmark with images from the Open Images dataset [17] to test models’ capability of describing novel objects which are not seen in the training corpus.”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “• What do I see? For object and people recognition, we use a VLM to rank OpenImages object categories (Kuznetsova et al. 2020) against the image, with the top m categories (out of 600) inserted into a second prefix: “Objects: {object1}, {object2}, {object3}.”
Pix2seq: A Language Modeling Framework for Object Detection (2021, methodology): “In common object detection datasets, such as Pascal VOC (Everingham et al., 2010), COCO (Lin et al., 2014), and OpenImages (Kuznetsova et al., 2020), images have variable numbers of objects, represented as sets of bounding boxes and class labels.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[123] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al.”
Reviving Iterative Training with Mask Guidance for Interactive Segmentation (2021, methodology): “We train the models for the following number of epochs on each dataset: 180 for ADE20k, 5 for OpenImages, 120 for SBD and Pascal VOC+SBD, 55 for LVIS and COCO+LVIS, 40 for COCO.”
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, methodology): “‡ used open images dataset (Kuznetsova et al., 2020) to expand its visual vocabulary of region feature embeddings.”
Large image datasets: A pyrrhic win for computer vision? (2020, background): “Yet, many of the efforts beyond ImageNet, including the Open Images dataset [50], have been built on top of the Creative commons loophole that large scale dataset curation agencies interpret as a free for all, consent-included green ﬂag.”
Rethinking Pre-training and Self-training (2020, methodology): “For selftraining, we use Open Images Dataset (OID) [60] as the self-training unlabeled data, which we found to be better than ImageNet (for more information about the effects of data sources on self-training, see Appendix F).”
Connecting Vision and Language with Localized Narratives (2019, methodology): “We annotated images from COCO [33, 9] and Open Images [31, 28].”
VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training (2020, methodology): “2009), Open Images (Kuznetsova et al. 2020), Objects365 (Shao et al.”
ImageNet-21K Pretraining for the Masses (2021, methodology): “For training, we adopted the high-quality training scheme described in [3], which was tested on other large-scale multi-label datasets, such as Open Images [28].”
Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI (2022, methodology): “Sampled from Open Images [20], the dataset included 100,000 bounding boxes over 30,000 images.”
Meshed-Memory Transformer for Image Captioning (2019, methodology): “Following [1], we use an object detector trained on Open Images 4 and ﬁlter detections by removing 39 Open Images classes that contain parts of objects or which are seldom mentioned.”