VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation

By Chuang Gan, Yandong Li, Haoxiang Li, Chen Sun, Boqing Gong. 2017

137 citations recorded by the source corpus.

Rich and dense human labeled datasets are among the main enabling factors for the recent advance on visionlanguage understanding. Many seemingly distant annotations (e.g., semantic segmentation and visual question answering (VQA)) are inherently connected in that they reveal different levels and perspectives of human understandings about the same visual scenes — and even the same set of images (e.g., of COCO). The popularity of COCO correlates those annotations and tasks. Explicitly linking them up may significantly benefit both individual tasks and the unified vision and language modeling.,,We present the preliminary work of linking the instance segmentations provided by COCO to the questions and answers (QAs) in the VQA dataset, and name the collected links visual questions and segmentation answers (VQS). They transfer human supervision between the previously separate tasks, offer more effective leverage to existing problems, and also open the door for new research problems and models. We study two applications of the VQS data in this paper: supervised attention for VQA and a novel question-focused semantic segmentation task. For the former, we obtain state-of-the-art results on the VQA real multiple-choice task by simply augmenting the multilayer perceptrons with some attention features that are learned using the segmentation-QA links as explicit supervision. To put the latter in perspective, we study two plausible methods and compare them to an oracle method assuming that the instance segmentations are given at the test stage.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “The existing visual/video question answering benchmarks [13, 50, 24, 40, 44] adopted images [1, 50, 11]/videos [40, 44, 19, 28, 18, 9, 45] and types of visual comprehension questions.”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, methodology): “Results are presented on the VQS dataset (Gan et al., 2017).”
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering (2020, background): “Visual question answering (VQA) [39,56,48,29,17] is a popular task for gauging the capability of visual reasoning systems.”
FindIt: Generalized Localization with Natural Language Queries (2022, background): “Large amounts of work have been done to learn across image and language, including visualgrounding [16,81,49,96,91,62], image captioning [6,1,5], visual question and answering (VQA) [33,12], visual dialog [11,39], visual reasoning [75,93,85], text-toimage and image-to-text retrieval [32,66,74], and ”
Referring Expression Comprehension via Cross-Level Multi-Modal Fusion (2022, background): “Referring expression comprehension (REC) [8, 35, 50, 51, 54] is a challenging task that aims at localizing the target object specified by a given referring expression, which can underpin a diverse set of vision-language tasks such as visual question answering [1, 13, 61], image captioning [39, 53], ”
Exploring Human-like Attention Supervision in Visual Question Answering (2017, methodology): “(Gan et al. 2017) proposed to use segmentations which are manually labeled by humans to supervise attention generated during training in VQA.”
Cascade Reasoning Network for Text-based Visual Question Answering (2020, background): “Compared with common VQA [12, 30], the text-based visual question answering (T-VQA) is more practical as it has the ability to aid visually impaired users in better understanding the information of their surrounding physical world, such as identifying time, temperatures and brand names [16].”
Learning compositionally through attentive guidance (2018, methodology): “This technique has been tried before in the context of VQA (Qiao, Dong, and Xu 2017; Gan et al. 2017) and Machine Translation (Mi, Wang, and Ittycheriah 2016).”
On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries (2020, unknown): “Attention and Alignments Explicit supervision for attention mechanisms (Bahdanau et al., 2015) is helpful for many tasks, including machine translation (Liu et al., 2016; Mi et al., 2016), image captioning (Liu et al., 2017), and visual question answering (Gan et al., 2017).”
Location-Aware Graph Convolutional Networks for Video Question Answering (2020, methodology): “Based on the visual sources, we can classify the VQA tasks into two categories: image QA (Goyal et al. 2017; Gan et al. 2017) and video QA (Lei et al. 2018; Yi et al. 2019).”
Are You Talking to Me? Reasoned Visual Dialog Generation Through Adversarial Learning (2017, background): “Visual dialog is the latest in a succession of vision-andlanguage problems that began with image captioning [9, 13, 36, 39], and includes visual question answering [1, 10, 40].”
Improving Question Answering over Incomplete KBs with Knowledge-Aware Reader (2019, unknown): “with symbolic approaches for visual QA (Gan et al., 2017; Mao et al., 2019; Hu et al., 2019).”
Compact Trilinear Interaction for Visual Question Answering (2019, background): “There are other works that consider answer information, besides image and question information, to improve VQA performance [16, 9, 36, 14, 41, 34].”
PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning (2021, background): “To assess machines’ ability on understanding and reasoning over vision and language, a wide variety of benchmarks have been created over the recent years [29, 3, 28, 16, 61, 12, 64, 36].”