All You May Need for VQA are Image Captions

By Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, Radu Soricut. 2022

93 citations recorded by the source corpus.

Visual Question Answering (VQA) has benefited from increasingly sophisticated models, but has not enjoyed the same level of engagement in terms of data creation. In this paper, we propose a method that automatically derives VQA examples at volume, by leveraging the abundance of existing image-caption annotations combined with neural models for textual question generation. We show that the resulting data is of high-quality. VQA models trained on our data improve state-of-the-art zero-shot accuracy by double digits and achieve a level of robustness that lacks in the same model trained on human-annotated VQA data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Neil Houlsby
3. Jianfeng Gao
4. Quoc V. Le
5. Yi Tay
6. Mostafa Dehghani
7. Percy Liang
8. Caiming Xiong

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Trevor Darrell
4. Cordelia Schmid
5. Jitendra Malik
6. Andrew Zisserman
7. Ming-Hsuan Yang
8. Xiaogang Wang

In Multimodal Learning

1. Steven C. H. Hoi
2. Junnan Li
3. Neil Houlsby
4. Radu Soricut
5. Soravit Changpinyo
6. Arsha Nagrani
7. Silvio Savarese
8. Matthias Minderer

In Question Answering & Reading Comprehension

1. Radu Soricut
2. Soravit Changpinyo
3. Yi Tay
4. Song-Chun Zhu
5. Arsha Nagrani
6. Hexiang Hu
7. Jialin Wu

Citations in the corpus

• PaLM-E: An Embodied Multimodal Language Model (2023, neutral): “Dataset in full mixture Sampling frequency % Webli (Chen et al., 2022) 100 52.4 VQ 2 A (Changpinyo et al., 2022) 25 13.1 VQG (Changpinyo et al., 2022) 10 5.2 CC3M (Sharma et al., 2018) 25 13.1 Object Aware (Piergiovanni et al., 2022) 10 5.2 OK-VQA (Marino et al., 2019) 1 0.5 VQAv2 (Goyal et al.,…”
• PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “English and Cross-Lingual VQA is VQ2ACC3M (Changpinyo et al., 2022a), translated in the same way as CC3M-35L.”
• Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022, methodology): “An orthogonal research direction for zero-shot VQA is to train the VLMs on synthetic VQA examples generated from captions (Changpinyo et al., 2022; Banerjee et al., 2021).”
• TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering (2023, methodology): “Inspired by prior work [4], given a text prompt T , we generate the question-answer tuples { Q i , C i , A i } Ni =1 via the pipeline illustrated in Figure 3.”
• PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “The pretraining mixture consists of the following data and objectives: (i) span corruption on textonly data (15% of tokens); (ii) split-captioning on WebLI alt-text data [21, 5]; (iii) captioning on CC3M [22] on native and translated alt-text data (over the same 35 languages covered by XM3600 [23]);”
• SQA3D: Situated Question Answering in 3D Scenes (2022, methodology): “Following prior practices that apply GPT-3 to VQA (Changpinyo et al., 2022; Gao et al., 2022), we propose to convert the 3D scene into text using an emerging technique called 3D captioning (Chen et al., 2021).”