VizWiz Grand Challenge: Answering Visual Questions from Blind People

By Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, Jeffrey P. Bigham. 2018

1,264 citations recorded by the source corpus.

The study of algorithms to automatically answer visual questions currently is motivated by visual question answering (VQA) datasets constructed in artificial VQA settings. We propose VizWiz, the first goal-oriented VQA dataset arising from a natural VQA setting. VizWiz consists of over 31,000 visual questions originating from blind people who each took a picture using a mobile phone and recorded a spoken question about it, together with 10 crowdsourced answers per visual question. VizWiz differs from the many existing VQA datasets because (1) images are captured by blind photographers and so are often poor quality, (2) questions are spoken and so are more conversational, and (3) often visual questions cannot be answered. Evaluation of modern algorithms for answering visual questions and deciding if a visual question is answerable reveals that VizWiz is a challenging dataset. We introduce this dataset to encourage a larger community to develop more generalized algorithms that can assist blind people.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “The ST-VQA dataset [8] contains 21,892 images from multiple sources including ICDAR 2013 [27], ICDAR 2015 [26], ImageNet [13], VizWiz [18], IIIT STR [39], Visual Genome [29], and COCO-Text [54].”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “In the past few years there has been a surge of interest in multimodal problems, from image captioning [8, 43, 88, 28, 68] to visual question answering (VQA) [3, 26, 27, 33, 71] and beyond.”
Towards VQA Models That Can Read (2019, methodology): “Despite collecting open-ended answers from annotators, we ﬁnd that human accuracy is 85.01%, consistent with that on VQA 2.0 [10] and VizWiz [13].”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “…of structured, task-based datasets [42,56], we also opt to include the training sets of various downstream tasks – adding the train sets of VQA [2], GQA [23], OKVQA [31], VizWiz [17], Hateful Memes [25], and CoCo Captions [9], resulting in a ﬁnal dataset of around 25 million image-text pairs.”
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, background): “VizWiz [13], although larger scale (31,173 mobile phone images contributed by 11,045 blind/low-vision users) targets image captioning and question-answering tasks, and is not annotated with object labels.”
Does Object Recognition Work for Everyone? (2019, background): “Recent advances in the accuracy of object-recognition systems [19, 20, 42] have spurred a large variety of realworld deployments of such systems, for instance, in aids for the visually impaired [17], in photo album organization software [45], in image search, and in popular cloud services [2, 8, 16,”
Improved Baselines with Visual Instruction Tuning (2023, methodology): “VizWiz [19] contains 8,000 images to evaluate model’s zero-shot generalization on visual questions asked by visually impaired people.”
Language Is Not All You Need: Aligning Perception with Language Models (2023, methodology): “Visual Question Answering Table 2b reports the visual question answering results on VQAv2 [41] and VizWiz [42].”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “On VQA, the evaluation benchmarks include VQAv2 [31], TextVQA [97], VizWiz-VQA [34].”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, methodology): “It is trained on Visual Genome [29] for object detection and attribute classification, and produces effective grid features for image VQA tasks [3, 20].”
In Defense of Grid Features for Visual Question Answering (2020, methodology): “Through a comprehensive set of experiments, we verified that our observations generalize across different network backbones, different VQA models [16, 49], different VQA benchmarks [3, 12], and even to other relevant tasks (e.”
CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks (2022, background): “Further, images in currently included datasets are sourced from social media, movie clips and web searches, thus excluding certain image domains, including those taken for accessibility needs such as descriptions for people with blindness [Gurari et al., 2018].”
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering (2018, background): “It has extensive applications in practice, such as assisting blind people [12] and education of young children, and therefore become a hot research topic recently.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, neutral): “TextVQA (Singh et al., 2019), VizWiz-QA (Gurari et al., 2018) and ST-VQA (Biten et al., 2019) require the ability to perform question answering in the presence of text in the input image.”
Scene Text Visual Question Answering (2019, methodology): “In total, we used six different datasets, namely: ICDAR 2013[23] and ICDAR2015[22], ImageNet [6], VizWiz[13], IIIT Scene Text Retrieval[37], Visual Genome [29] and COCO-Text [50].”
Experts Collaboration Learning for Continual Multi-Modal Reasoning (2023, background): “life of visually-impaired people [13], and driving assistance systems in assisting drivers to avoid accidents [2].”
"Person, Shoes, Tree. Is the Person Naked?" What People with Vision Impairments Want in Image Descriptions (2020, background): “These findings affirm the need for further research related to next-generation, interactive technologies for describing images [3, 19, 28].”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, methodology): “, 2017), VizWiz (Gurari et al., 2018), and ParsVQA-Caps (Mobasher et al.”