Active Learning for Visual Question Answering: An Empirical Study

By Xiaoyu Lin, Devi Parikh. 2017

35 citations recorded by the source corpus.

We present an empirical study of active learning for Visual Question Answering, where a deep VQA model selects informative question-image pairs from a pool and queries an oracle for answers to maximally improve its performance under a limited query budget. Drawing analogies from human learning, we explore cramming (entropy), curiosity-driven (expected model change), and goal-driven (expected error reduction) active learning approaches, and propose a fast and effective goal-driven active learning scoring function to pick question-image pairs for deep VQA models under the Bayesian Neural Network framework. We find that deep VQA models need large amounts of training data before they can start asking informative questions. But once they do, all three approaches outperform the random selection baseline and achieve significant query savings. For the scenario where the model is allowed to ask generic questions about images but is evaluated only on specific questions (e.g., questions whose answer is either yes or no), our proposed goal-driven scoring function performs the best.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “The sample selection problem is also closely related to active learning strategies (Cohn, Atlas, & Ladner, 1994; Lin & Parikh, 2017; Wang, Zhang, Li, Zhang, & Lin, 2016; Settles, 2009; Yoo & So Kweon, 2019; Wei, Iyer, & Bilmes, 2015), with the most common selection methods using uncertainty…”
A Survey of Deep Active Learning (2020, methodology): “DAL has been widely utilized in various fields, including image recognition [35, 47, 53, 68], text classification [145, 180, 185], visual question answering [98] and object detection [3, 39, 121], etc.”
What Do I Annotate Next? An Empirical Study of Active Learning for Action Localization (2018, background): “Very recently, active learning approaches have emerged in more contemporary vision tasks, including human pose estimation [46] and visual question answering [45].”