Are You Talking to Me? Reasoned Visual Dialog Generation Through Adversarial Learning
By Qi Wu, Peng Wang, Chunhua Shen, Ian David Reid, Anton van den Hengel. 2017
133 citations recorded by the source corpus.
The visual dialog task requires an agent to engage in a conversation about an image with a human. It represents an extension of the visual question answering task in that the agent needs to answer a question about an image, but it needs to do so in light of the previous dialog that has taken place. The key challenge in visual dialog is thus maintaining a consistent, and natural dialog while continuing to answer questions correctly. We present a novel approach that combines Reinforcement Learning and Generative Adversarial Networks (GANS) to generate more human-like responses to questions. The GAN helps overcome the relative paucity of training data, and the tendency of the typical MLE-based approach to generate overly terse answers. Critically, the GAN is tightly integrated into the attention mechanism that generates human-interpretable reasons for each answer. This means that the discriminative model of the GAN has the task of assessing whether a candidate answer is generated by a human or not, given the provided reason. This is significant because it drives the generative model to produce high quality answers that are well supported by the associated reasoning. The method also generates the state-of-the-art results on the primary benchmark.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Cordelia Schmid
- Yi Yang
- Jitendra Malik
In Multimodal Learning
In Dialogue & Conversational AI
- Hanwang Zhang
- Takayuki Okatani
- Van-Quang Nguyen
- Masanori Suganuma
- Can Xu
- Daxin Jiang
- Xiuyi Chen
- Zhengjun Zha
Citations in the corpus
- Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, background): “Follow-up works on visual dialog have explored the use of deep reinforcement learning [3, 4, 17], knowledge transfer from discriminative to generative decoders [5], conditional variational autoencoders [6], generative adversarial networks [7], attention mechanisms for visual coreference resolution [”
- Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, methodology): “Generative Adversarial Network [16, 55, 29] was used in [51] to generate answers indistinguishable from human-generated answers, and a conditional variational au-toencoder [25, 43] was developed in [33] to promote answer diversity.”
- Efficient Attention Mechanism for Handling All the Interactions between Many Inputs with Application to Visual Dialog (2019, methodology): “Existing studies have considered attention from one utility to another based on different hypotheses, such as “question→ history→ image” path in [19, 27], and “question→ image → history → question” path in [13, 40], etc.”
- Image-Question-Answer Synergistic Network for Visual Dialog (2019, methodology): “Generative models usually employ seq2seq [24] or advanced reinforcement learning [29] techniques to generate the answer set, where the highest probability answer is cho-”
- Visual Grounding Via Accumulated Attention (2020, background): “As a research direction across vision and language, VG has benefited from the development of Convolutional Neu-7746 ral Networks [7, 12], Recurrent Neural Networks [2, 6], and other areas such as Image Captioning [5, 26] and Visual Question Answering (VQA) [27].”
- DialGraph: Sparse Graph Learning Networks for Visual Dialog (2020, methodology): “…on visual dialog into three groups: (1) soft attention-based methods that compute attended representations of the image, and the history [8,28,42,13,9,38,30], (2) a visual coreference resolution [39,25,32,19] that clarifies ambiguous expressions ( e.g ., it, them) in the question and links…”
- Multi-View Attention Networks for Visual Dialog (2020, methodology): “…following methods: LF (Das et al., 2017), HRE (Das et al., 2017), MN (Das et al., 2017), HCIAE (Lu et al., 2017), AMEM (Seo et al., 2017), CoAtt (Wu et al., 2018), FGA (Schwartz et al., 2019), CorefNMN (Kottur et al., 2018), RVA (Niu et al., 2019), DualVD (Jiang et al., 2020), CAG (Guo et al.,…”
- Dual Attention Networks for Visual Reference Resolution in Visual Dialog (2019, background): “…approaches (LF and HRE [Das et al. , 2017]), (2) Attention-based approaches (MN [Das et al. , 2017], HCIAE [Lu et al. , 2017] and CoAtt [Wu et al. , 2018]), and (3) Approaches that deal with visual reference resolution in VisDial (AMEM [Seo et al. , 2017], CorefNMN [Kottur et al. ,…”
- Reasoning Visual Dialogs With Structural and Partial Observations (2019, background): “Quantitative Results: We compare our method with several state-of-the-art discriminative dialog models, i.e., LF [9], HRE [9], HREA [9], MN [9], SAN-QI [64], HieCoAtt-QI [36], AMEM [51], HCIAE-NP-ATT [34], SF [22], and SCA [62].”
- Recursive Visual Attention in Visual Dialog (2018, methodology): “The attention-based methods include HREA [9], MN [9], HCIAE [22], and CoAtt [37].”
- Iterative Context-Aware Graph Inference for Visual Dialog (2020, methodology): “To improve performance, various attention mechanisms have been widely used in the task, including history-conditioned image attention (HCIAE) [23], sequential co-attention (CoAtt) [34], dual visual attention (DVAN) [9], and recurrent dual attention (ReDAN) [7].”
- See More, Know More: Unsupervised Video Object Segmentation With Co-Attention Siamese Networks (2019, background): “More recently, co-attention mechanisms have been studied in vision-and-language tasks, such as visual question answering [35, 65, 63, 39] and visual dialogue [63].”
- Dual Visual Attention Network for Visual Dialog (2019, methodology): “CoAtt-RL [Wu et al. , 2018], which was pre-trained with the MLE loss, adopted both reinforcement learning and adversarial learning for further offline training.”
- Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation (2020, background): “Further, co-attention mechanisms become an essential tool in many vision-language applications and sequential modeling tasks, such as visual question answering [39,42,76,79], visual dialog [74,85], vision-language navigation [66], and video segmentation [40,60], showing its effectiveness in capturin”
- Modality-Balanced Models for Visual Dialogue (2020, background): “Niu et al. (2018) tries to extract the clues from history recursively while Wu et al. (2018) and Guo, Xu, and Tao (2019) employ co-attention to fuse visual, history, and question features.”