Separating Skills and Concepts for Novel Visual Question Answering

By Spencer Whitehead, Hui Wu, Heng Ji, Rogério Schmidt Feris, Kate Saenko, Uiuc MIT-IBM. 2021

40 citations recorded by the source corpus.

Generalization to out-of-distribution data has been a problem for Visual Question Answering (VQA) models. To measure generalization to novel questions, we propose to separate them into "skills" and "concepts". "Skills" are visual tasks, such as counting or attribute recognition, and are applied to "concepts" mentioned in the question, such as objects and people. VQA methods should be able to compose skills and concepts in novel ways, regardless of whether the specific composition has been seen in training, yet we demonstrate that existing models have much to improve upon towards handling new compositions. We present a novel method for learning to compose skills and concepts that separates these two factors implicitly within a model by learning grounded concept representations and disentangling the encoding of skills from that of concepts. We enforce these properties with a novel contrastive learning procedure that does not rely on external annotations and can be learned from unlabeled image-question pairs. Experiments demonstrate the effectiveness of our approach for improving compositional and grounding performance.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

ViperGPT: Visual Inference via Python Execution for Reasoning (2023, neutral): “This visual reasoning procedure has been explored by a variety of works [29, 57].”
DALL-EVAL: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models (2022, background): “[66], we define three visual reasoning skills: object recognition, object counting, and spatial relation understanding.”
Webly Supervised Concept Expansion for General Purpose Vision Models (2022, background): “[82] disentangle the encoding of concepts and skills and build a model that can generalize to new skill-concept compositions and new concepts for VQA.”
Towards General Purpose Vision Systems: An End-to-End Task-Agnostic Vision-Language Architecture (2021, background): “Other works focus on unidirectional transfer to a single task such as captioning [15] or VQA [50].”
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers (2022, methodology): “Inspired by the skill-concept analysis for visual question answering [61], we de-ﬁne four visual reasoning skills to evaluate: object recognition, object counting, color recognition, and spatial relation understanding.”