Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
By Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma and 2 more. 2016
6,638 citations recorded by the source corpus.
Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in an image. When asked “What vehicle is the person riding?”, computers will need to identify the objects in an image as well as the relationships riding(man, carriage) and pulling(horse, carriage) to answer correctly that “the person is riding a horse-drawn carriage.” In this paper, we present the Visual Genome dataset to enable the modeling of such relationships. We collect dense annotations of objects, attributes, and relationships within each image to learn these models. Specifically, our dataset contains over 108K images where each image has an average of 35\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$35$$\end{document} objects, 26\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$26$$\end{document} attributes, and 21\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$21$$\end{document} pairwise relationships between objects. We canonicalize the objects, attributes, relationships, and noun phrases in region descriptions and questions answer pairs to WordNet synsets. Together, these annotations represent the densest and largest dataset of image descriptions, objects, attributes, relationships, and question answer pairs.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Multimodal Learning
In Evaluation & Benchmarks
- Mohit Bansal
- Yejin Choi
- Jianfeng Gao
- Li Fei-Fei
- Trevor Darrell
- Abhinav Kumar Gupta
- Jitendra Malik
- Ali Farhadi
Citations in the corpus
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “While MS-COCO and Visual Genome are high quality crowd-labeled datasets, they are small by modern standards with approximately 100,000 training photos each.”
- Experience Grounds Language (2020, methodology): “We posit that models must be able to watch and recognize objects, people, and activities to understand the language describing them (Li et al., 2019b; Krishna et al., 2017; Yatskar et al., 2016; Perlis, 2016).”
- PIQA: Reasoning about Physical Commonsense in Natural Language (2019, background): “Research has studied predicting visual relationships in images (Krishna et al. 2016) and as well as actions and their dependent objects (Yatskar, Zettlemoyer, and Farhadi 2016).”
- UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “We construct our pre-training dataset based on four existing V+L datasets: COCO (Lin et al., 2014), Visual Genome (VG) (Krishna et al., 2017), Conceptual Captions (CC) (Sharma et al., 2018), and SBU Captions (Ordonez et al., 2011 purpose, which makes the model framework more scalable, as additional…”
- VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “For instance, attribute annotations in Visual Genome (Krishna et al., 2017) are used to enhance the object detector in VQA systems (Anderson et al., 2018).”
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, methodology): “We use Faster R-CNN [31] (with ResNet-101 [11] backbone) pretrained on the Visual Genome dataset [16] (see [30] for details) to extract region features.”
- A Multimodal Framework for the Detection of Hateful Memes (2020, methodology): “The large-scale pretraining is performed over four V+L datasets – COCO, Visual Genome, CC, and SBU Captions [26].”
- ALICE: Active Learning with Contrastive Natural Language Explanations (2020, neutral): “The off-theshelf semantic segment localization models could be the pre-trained localization models on various large-scale datasets like Visual Genome (Krishna et al., 2017) and PASCAL-Part (Chen et al., 2014).”
- ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “And for fair comparison with the models pretrained on both out-of-domain and in-domain datasets, we further pretrain ERNIE-ViL on MS-COCO [18] and Visual-Genome [19] (in-domain datasets for downstream tasks).”
- VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “As described in Section 2, all these methods use an object detector trained on Visual Genome [73] (with ImageNet-pretrained backbone) to extract visual features, made available by [74].”
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “st cross-attention, then self-attention). Implementation Details For UNITER experiments, we pre-train with the same four large-scale datasets used in the original model: COCO [36], Visual Genome (VG) [28], Conceptual Captions [58] and SBU Captions [49]. VILLA is applied to both MLM and ITM pre-train”
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, methodology): “In the past few years there has been a surge of interest in multimodal problems, from image captioning [8, 43, 88, 28, 68] to visual question answering (VQA) [3, 26, 27, 33, 71] and beyond.”
- ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “UNITER[21] combines four datasets (Conceptual Captions[2], SBU Captions[3], Visual Genome[22] and MSCOCO[5]) together to form a 9.”
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, background): “XLM (Lample and Conneau, 2019) learns the joint cross-lingual representations by leveraging the monolingual data and parallel data.”
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “UNITER add about 1 million image-caption pair besides the Conceptual Captions and in-domain MSCOCO Caption and Visual Genome Dense Captions (Krishna et al. 2017) data.”
- Towards VQA Models That Can Read (2019, methodology): “The region based features are extracted from fc6 layer of an improved Faster-RCNN model [8] trained on the Visual Genome [28] objects and attributes as provided in [1].”