Stacking with Auxiliary Features for Visual Question Answering

By Nazneen Rajani, Raymond J. Mooney. 2018

17 citations recorded by the source corpus.

Visual Question Answering (VQA) is a well-known and challenging task that requires systems to jointly reason about natural language and vision. Deep learning models in various forms have been the standard for solving VQA. However, some of these VQA models are better at certain types of image-question pairs than other models. Ensembling VQA models intelligently to leverage their diverse expertise is, therefore, advantageous. Stacking With Auxiliary Features (SWAF) is an intelligent ensembling technique which learns to combine the results of multiple models using features of the current problem as context. We propose four categories of auxiliary features for ensembling for VQA. Three out of the four categories of features can be inferred from an image-question pair and do not require querying the component models. The fourth category of auxiliary features uses model-specific explanations. In this paper, we describe how we use these various categories of auxiliary features to improve performance for VQA. Using SWAF to effectively ensemble three recent systems, we obtain a new state-of-the-art. Our work also highlights the advantages of explainable AI models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Explain Yourself! Leveraging Language Models for Commonsense Reasoning (2019, methodology): “In the multi-modal setting, Rajani and Mooney (2018) showed that visual explanations can be leveraged to improve performance of VQA (An-tol et al., 2015) and that an ensemble explanation is signiﬁcantly better than individual explanations using both automated and human evaluations (Ra-jani and…”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “The so-called Stacking With Auxiliary Features (SWAF) approach proposed in [242] points in this direction by harnessing and integrating explanations in stacking ensembles to improve their generalization.”
The Unreliability of Explanations in Few-Shot In-Context Learning (2022, methodology): “Regarding our use of calibration, similar ideas of explanation-based performance estimation have been applied to other tasks (Rajani and Mooney, 2018; Ye et al., 2021; Ye and Durrett, 2022), but we rely on the free-text explanations generated by the model instead of interpretations obtained through…”
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning (2022, methodology): “Regarding our use of calibration, similar ideas of explanation-based performance estimation have been applied to other tasks (Rajani and Mooney, 2018; Ye et al., 2021; Ye and Durrett, 2022), but we rely on the free-text explanations generated by the model instead of interpretations obtained through ”