Learning by Asking Questions

By Ishan Misra, Ross B. Girshick, Rob Fergus, Martial Hebert, Abhinav Kumar Gupta, L. V. D. van der Maaten. 2017

86 citations recorded by the source corpus.

We introduce an interactive learning framework for the development and testing of intelligent visual systems, called learning-by-asking (LBA). We explore LBA in context of the Visual Question Answering (VQA) task. LBA differs from standard VQA training in that most questions are not observed during training time, and the learner must ask questions it wants answers to. Thus, LBA more closely mimics natural learning and has the potential to be more data-efficient than the traditional VQA setting. We present a model that performs LBA on the CLEVR dataset, and show that it automatically discovers an easy-to-hard curriculum when learning interactively from an oracle. Our LBA generated data consistently matches or outperforms the CLEVR train data and is more sample efficient. We also show that our model asks questions that generalize to state-of-the-art VQA models and to novel test time distributions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “There have also been numerous visual reasoning models (Hudson & Manning, 2018; Santoro et al., 2017; Hu et al., 2017; Perez et al., 2018; Zhu et al., 2017; Mascharka et al., 2018; Suarez et al., 2018; Cao et al., 2018; Bisk et al., 2018; Misra et al., 2018; Aditya et al., 2018).”
REMIND Your Neural Network to Prevent Catastrophic Forgetting (2019, methodology): “In [62], a model was developed to improve VQA abilities by asking questions; however, the”
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding (2018, background): “From a broader perspective, Misra et al. [2017] explored learning to reason by asking questions and Bisk et al. [2018] studied spatial reasoning in a 3D blocks world.”
Cycle-Consistency for Robust Visual Question Answering (2019, background): “Training models to ask informative questions about an image in an active learning ﬁxed-budget setting was explored in [31].”
Learning to Caption Images Through a Lifetime by Asking Questions (2018, background): “Furthermore, [18] operates in CLEVR [8], a synthetic environment and questions are limited to programs rather than natural language.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, unknown): “Also, in the context of interpretable learning framework learning-by-asking (LBA) (Misra et al., 2018b) is proposed to closely mimic natural learning with the goal to make it more data efficient than the traditional VQA setting.”
Challenges and Prospects in Vision and Language Research (2019, background): “, humans or databases, and then they incorporate the new information for future use (Misra et al., 2018; Yang et al., 2018; Zhang et al., 2018a).”
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering (2020, methodology): “Models 70k 140k 280k 420k 630k 700k FiLM-LBA [42] 51.”
Semantic Curiosity for Active Visual Learning (2020, unknown): “This has been used in a number of applications such as medical image analysis [30], training object detectors [50,55], video segmentation [21], and visual question answering [36].”
Why Build an Assistant in Minecraft? (2019, methodology): “This has been studied in (ungrounded) conversational systems [77, 90, 68, 49] and when grounding with images via the CLEVR QA task [62].”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, background): “The end-to-end methods [38, 81, 72, 58] typically tackle the visual question answering (VQA) problem by designing monolithic multi-modal deep networks [30, 33].”