A Survey of Current Datasets for Vision and Language Research

By Francis Ferraro, Nasrin Mostafazadeh, Ting-Hao 'Kenneth' Huang, Lucy Vanderwende, Jacob Devlin, Michel Galley, Margaret Mitchell. 2015

77 citations recorded by the source corpus.

Integrating vision and language has long been a dream in work on artificial intelligence (AI). In the past two years, we have witnessed an explosion of work that brings together vision and language from images to videos and beyond. The available corpora have played a crucial role in advancing this area of research. In this paper, we propose a set of quality metrics for evaluating and analyzing the vision & language datasets and categorize them accordingly. Our analyses show that the most recent datasets have been using more complex language and more abstract concepts, however, there are different strengths and weaknesses in each.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

Measuring Data (2022, background): “Examples from research on text and image data include “quality criteria” to analyze, evaluate, and compare the quality of vision and language datasets against one another (Ferraro et al., 2015); “task-independent metrics” to reflect states of the data without the contextual knowledge of the applicat”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “, 2016) focus mostly on identification of object properties and few spatial relations (Section 4; Ferraro et al., 2015; Alikhani and Stone, 2019).”
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering (2017, background): “Despite its short history, there has been a flourishing amount of research produced within the past few years [9, 6].”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “5The resource provided by Ferraro et al. (2015) can be found at http:// visionandlanguage.”
Visual Storytelling (2016, methodology): “5 Table 2: A summary of our dataset, following the proposed analyses of Ferraro et al. (2015), including the Frazier and Yngve measures of syntactic complexity.”
Visual question answering: A survey of methods and datasets (2016, background): “Those datasets go beyond the scope of this article (see [19] for a review), but are a potential source of additional training data for VQA since they combine images with textual annotations.”
Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos (2018, background): “Connecting vision and language has emerged as a prominent multi-disciplinary research problem [11].”
Visual to Text: Survey of Image and Video Captioning (2019, background): “Please refer to [117] for a recent survey that discusses the datasets for vision and language re-”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Surveys which went beyond a specific task have summarized dataset statistics (Ferraro et al., 2015), provided comprehensive overview of only natural language processing tasks such as natural language generation (NLG) (Gatt & Krahmer, 2018) and commonsense reasoning (Storks et al.”
Generating Natural Questions About an Image (2016, unknown): “Is this a large doughnut?”
Measuring the Diversity of Automatic Image Descriptions (2018, background): “Ferraro et al. (2015) present other metrics in their survey of datasets for vision and language research, including: Yngve and Frazier measurements of syntactic complexity (Yngve, 1960; Frazier, 1985).”