Improving Generative Visual Dialog by Answering Diverse Questions
By Vishvak Murahari, Prithvijit Chattopadhyay, Dhruv Batra, Devi Parikh, Abhishek Das. 2019
38 citations recorded by the source corpus.
Prior work on training generative Visual Dialog models with reinforcement learning ((Das et al., ICCV 2017) has explored a Q-Bot-A-Bot image-guessing game and shown that this ‘self-talk’ approach can lead to improved performance at the downstream dialog-conditioned image-guessing task. However, this improvement saturates and starts degrading after a few rounds of interaction, and does not lead to a better Visual Dialog model. We find that this is due in part to repeated interactions between Q-Bot and A-BOT during self-talk, which are not informative with respect to the image. To improve this, we devise a simple auxiliary objective that incentivizes Q-Bot to ask diverse questions, thus reducing repetitions and in turn enabling A-Bot to explore a larger state space during RL i.e. be exposed to more visual concepts to talk about, and varied questions to answer. We evaluate our approach via a host of automatic metrics and human studies, and demonstrate that it leads to better dialog, i.e. dialog that is more diverse (i.e. less repetitive), consistent (i.e. has fewer conflicting exchanges), fluent (i.e., more human-like), and detailed, while still being comparably image-relevant as prior work and ablations.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dhruv Batra
- Devi Parikh
- Abhishek Das
- Mohit Bansal
- Jianfeng Gao
- Hanwang Zhang
- Vishvak Murahari
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Dhruv Batra
- Devi Parikh
- Trevor Darrell
- Luc Van Gool
- Dacheng Tao
- Abhishek Das
- Kristen Grauman
- Chunhua Shen
In Question Answering & Reading Comprehension
- Devi Parikh
- Dhruv Batra
- Nazneen Fatema Rajani and Raymond J. Mooney
- Christopher D. Manning
- Drew A. Hudson
- Song-Chun Zhu
- Quanshi Zhang
- Raymond J. Mooney
Citations in the corpus
- Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, background): “Follow-up works on visual dialog have explored the use of deep reinforcement learning [3, 4, 17], knowledge transfer from discriminative to generative decoders [5], conditional variational autoencoders [6], generative adversarial networks [7], attention mechanisms for visual coreference resolution [”