Learning to Caption Images Through a Lifetime by Asking Questions

By Tingke Shen, Amlan Kar, Sanja Fidler. 2018

33 citations recorded by the source corpus.

In order to bring artificial agents into our lives, we will need to go beyond supervised learning on closed datasets to having the ability to continuously expand knowledge. Inspired by a student learning in a classroom, we present an agent that can continuously learn by posing natural language questions to humans. Our agent is composed of three interacting modules, one that performs captioning, another that generates questions and a decision maker that learns when to ask questions by implicitly reasoning about the uncertainty of the agent and expertise of the teacher. As compared to current active learning methods which query images for full captions, our agent is able to ask pointed questions to improve the generated captions. The agent trains on the improved captions, expanding its knowledge. We show that our approach achieves better performance using less human supervision than the baselines on the challenging MSCOCO dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Few-Shot Continual Active Learning by a Robot (2022, background): “For object learning, many researchers have presented active learning techniques using uncertainty sampling [35, 18, 14, 15, 36, 16].”
Unsupervised Commonsense Question Answering with Self-Talk (2020, background): “Shen et al. (2019) developed an active learning framework for image captioning that learns to detect uncertainty about generated words and ask natural language questions to reduce its uncertainty.”
Learning Novel Objects Continually Through Curiosity (2021, background): “In recent years, attempts have been made to develop active learning techniques to reduce the cost of labelling [7], [8], [9], [10], [11].”
Online Learning of Objects through Curiosity-Driven Active Learning (2021, background): “In recent years, attempts have been made to develop active learning techniques to reduce the cost of labelling [7], [8], [9], [10], [11].”
From Show to Tell: A Survey on Image Captioning (2021, background): “In particular, when a reinforcement learning phase is performed, rewards based on human feedback or interaction can be considered [217].”