Captioning Images Taken by People Who Are Blind

By Danna Gurari, Yinan Zhao, Meng Zhang, Nilavra Bhattacharya. 2020

219 citations recorded by the source corpus.

While an important problem in the vision community is to design algorithms that can automatically caption images, few publicly-available datasets for algorithm development directly address the interests of real users. Observing that people who are blind have relied on (human-based) image captioning services to learn about images they take for nearly a decade, we introduce the first image captioning dataset to represent this real use case. This new dataset, which we call VizWiz-Captions, consists of over 39,000 images originating from people who are blind that are each paired with five captions. We analyze this dataset to (1) characterize the typical captions, (2) characterize the diversity of content found in the images, and (3) compare its content to that found in eight popular vision datasets. We also analyze modern image captioning algorithms to identify what makes this new dataset challenging for the vision community. We publicly-share the dataset with captioning challenge instructions at this https URL

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “In the past few years there has been a surge of interest in multimodal problems, from image captioning [8, 43, 88, 28, 68] to visual question answering (VQA) [3, 26, 27, 33, 71] and beyond.”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “The number of images in these datasets range from fewer than a few hundred thousands [79, 20, 41, 27] to several millions [66], with lower text quality as the scale increases.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “We comprehensively evaluate the captioning performance on the widely-used Karpathy split (Karpathy & Li, 2015) of COCO (Lin et al., 2014) and Flickr30K (Young et al., 2014), the COCO test set, nocaps (Agrawal et al., 2019)2 which focuses on novel objects, TextCaps (Sidorov et al., 2020) which focuse”
Multimodal datasets: misogyny, pornography, and malignant stereotypes (2021, background): “This has attracted the attention of accessibility advocates and ethicists alike, who have created a robust body of work that critically analyzes the Image-Textual-description dyad [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27].”
TextCaps: a Dataset for Image Captioning with Reading Comprehension (2020, background): “In concurrent work, [15] collect captions on VizWiz [5] images but unlike TextCaps there isn’t a specific focus on reading comprehension.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, background): “VizWiz-Cap (Gurari et al., 2020) contains images taken by people who are blind, which also involves scene-text understanding.”
Grounding Answers for Visual Questions Asked by Visually Impaired People (2022, methodology): “For quality issue severity, we follow the process described in [16] to divide the VQAs into three buckets of “None”,“Some” and “Severe” based on their image quality.”
Concadia: Tackling image accessibility with context (2021, background): “…in deep learning have led to a surge of models that create text from images (e.g., Karpathy and Fei-Fei, 2015; Xu et al., 2016; Lu et al., 2017; Anderson et al., 2018; Biten et al., 2019) and a variety of datasets to train them on (Lin et al., 2014; Sharma et al., 2018; Gurari et al., 2020).”
PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “Notably, VizWiz-Cap and Widget-Cap shows improved performance from multitask fine-tuning.”
TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text (2021, unknown): “VizWiz-Captions [15] also contains similar captions on VizWiz images [3] but doesn’t explicitly require scene-text reasoning.”
Multimodal Large Language Models: A Survey (2023, background): “The audio modality can capture changes in the speaker’s tone and speech rate; the visual modality can analyze facial expressions and body language; and the ECG and EEG can provide physiological signals related to emotional changes.”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “impaired [235], where image captioning approaches must be able to provide informative descriptions even for low-quality visual inputs [132].”