Visual Madlibs: Fill in the blank Image Generation and Question Answering

By Licheng Yu, Eunbyung Park, Alexander C. Berg, Tamara L. Berg. 2015

98 citations recorded by the source corpus.

In this paper, we introduce a new dataset consisting of 360,001 focused natural language descriptions for 10,738 images. This dataset, the Visual Madlibs dataset, is collected using automatically produced fill-in-the-blank templates designed to gather targeted descriptions about: people and objects, their appearances, activities, and interactions, as well as inferences about the general scene or its broader context. We provide several analyses of the Visual Madlibs dataset and demonstrate its applicability to two new description generation tasks: focused description generation, and multiple-choice question-answering for images. Experiments using joint-embedding and deep learning methods show promising results on these tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “The Visual Madlibs dataset has 20k image cap-tions about hypothetical next/previous events (Yu et al., 2015); similar to our work, the test portion is multiple-choice, with counterfactual answers retrieved from similar images and veriﬁed by humans.”
VizWiz Grand Challenge: Answering Visual Questions from Blind People (2018, methodology): “Over the past three years, a plethora of VQA datasets have been publicly shared to encourage a larger community to collaborate on developing algorithms that answer visual questions [6, 7, 16, 17, 19, 20, 23, 28, 32, 39, 40, 43, 45].”
Visual Dialog (2016, background): “A number of problems at the intersection of vision and language have recently gained prominence – image captioning [10, 11, 20, 54], video/movie description [43, 51, 52], text-to-image coreference/grounding [16, 22, 37, 39, 42], Visual Madlibs [60], and of course, visual question answering (VQA) [4,”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “This line of work also includes ‘pointing’ questions [45, 93] and templated questions with open ended answers [86].”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, methodology): “…et al. 2014; Klein et al. 2014; Kulkarni et al. 2011; Lebret et al. 2015; Mao et al. 2015b; Ordonez et al. 2011; Vinyals et al. 2015; Yao et al. 2010) and visual question answering (Antol et al. 2015; Gao et al. 2015; Krishna et al. 2016; Malinowski and Fritz 2014; Ren et al. 2015; Yu et al. 2015).”
Neural Module Networks (2015, methodology): “The visual question answering task has significant significant applications to human-robot interaction, search, and accessibility, and has been the subject of a great deal of recent research attention [3, 10, 26, 28, 33, 40].”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, background): “There is a surge of interest in image-text tasks such as image captioning [10, 22, 23, 25, 31, 43, 46, 50] and visual question answering [2, 12, 52].”
TVQA: Localized, Compositional Video Question Answering (2018, background): “Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al.”
MovieQA: Understanding Stories in Movies through Question-Answering (2015, methodology): “For QA, [23] provides questions and answers (mainly lists of objects, colors, etc.) for the NYUv2 RGB-D dataset, while [1, 47] do so for MS-COCO with a dataset of a million QAs.”
Ask Your Neurons: A Neural-Based Approach to Answering Questions about Images (2015, background): “Also several new datasets for Visual Turing Tests have just been proposed [1, 35] that are worth further investigations.”
Where to Look: Focus Regions for Visual Question Answering (2015, background): “Ba et al. [2] also uses a similar architecture, but in a zero-shot learning framework to predict classiﬁers for novel categories.”
A Dataset and Exploration of Models for Understanding Video Data through Fill-in-the-Blank Question-Answering (2016, methodology): “In [43], the authors take a similar approach to ours by transforming the description task into fill-in-the-blank questions about images.”
Explicit Knowledge-based Reasoning for Visual Question Answering (2015, methodology): “Several VQA datasets [Malinowski and Fritz, 2014b; Ren et al., 2015; Antol et al., 2015; Yu et al., 2015; Krishna et al., 2017] have also been constructed for making and measuring progress.”
Learning Two-Branch Neural Networks for Image-Text Matching Tasks (2017, background): “Motivated by the notion of creating a visual Turing test, Visual Question Answering (VQA) [7], [8], [9] aims at answering freeform questions about image content.”
Neural Motifs: Scene Graph Parsing with Global Context (2017, background): “Deep sequential models have demonstrated strong performance for tasks such as captioning [6, 9, 40, 18] and visual question answer [1, 34, 48, 12, 8], including for problems not classically thought of as sequence problems, such as language parsing [39], or multi-label classiﬁcation [41].”
Detecting and Recognizing Human-Object Interactions (2017, background): “Accurate recognition of human-object interaction can benefit numerous tasks in computer vision, such as actionspecific image retrieval [26], caption generation [35], and question answering [35, 24].”
Uncovering the Temporal Context for Video Question Answering (2017, methodology): “Following Yu et al. (2015), we also use the video captioning model to generate the description of the video.”