Image-Question-Answer Synergistic Network for Visual Dialog

By Dalu Guo, Chang Xu, Dacheng Tao. 2019

83 citations recorded by the source corpus.

The image, question (combined with the history for de-referencing), and the corresponding answer are three vital components of visual dialog. Classical visual dialog systems integrate the image, question, and history to search for or generate the best matched answer, and so, this approach significantly ignores the role of the answer. In this paper, we devise a novel image-question-answer synergistic network to value the role of the answer for precise visual dialog. We extend the traditional one-stage solution to a two-stage solution. In the first stage, candidate answers are coarsely scored according to their relevance to the image and question pair. Afterward, in the second stage, answers with high probability of being correct are re-ranked by synergizing with image and question. On the Visual Dialog v1.0 dataset, the proposed synergistic network boosts the discriminative visual dialog model to achieve a new state-of-the-art of 57.88% normalized discounted cumulative gain. A generative visual dialog model equipped with the proposed technique also shows promising improvements.

Subfields: Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

In Dialogue & Conversational AI

Citations in the corpus

VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, methodology): “, 2019), Synergistic (Guo et al., 2019a), DAN (Kang et al.”
DialGraph: Sparse Graph Learning Networks for Visual Dialog (2020, background): “…on visual dialog into three groups: (1) soft attention-based methods that compute attended representations of the image, and the history [8,28,42,13,9,38,30], (2) a visual coreference resolution [39,25,32,19] that clariﬁes ambiguous expressions ( e.g ., it, them) in the question and links them…”
Making History Matter: History-Advantage Sequence Training for Visual Dialog (2019, methodology): “The proposed approaches for Visual Dialog are based on encoder-decoder structure, and can be categorized into three groups based on the design of encoder: (1) Fusion-based Models (LF [6], HRE [6], Sync [11]), the methods fuse image, question, and history features at different stages.”
Bilinear Graph Networks for Visual Question Answering (2019, background): “image captioning [1], visual grounding [37, 8], visual question answering (VQA) [2, 9, 34], and visual dialog [6, 12].”
Recursive Visual Attention in Visual Dialog (2018, background): “0 leaderboard1, while the team DL-61 [13] has achieved the best NDCG record 0.”
Efficient Attention Mechanism for Handling All the Interactions between Many Inputs with Application to Visual Dialog (2019, methodology): “0 datasets, which include LF, HRE, MN [8], LF-Att, MN-Att (with attention) [8], SAN [42], AMEM [35], SF [18], HCIAE [27] and Sequential CoAtten-tion model (CoAtt) [40], Synergistic [15], FGA [34], GNN [48], RvA [30], CorefNMN [22], HCAN [41], DAN [19], and ReDAN [13].”
Multi-View Attention Networks for Visual Dialog (2020, methodology): “0 dataset for the following methods: Late Fusion (LF) [7], Hierarchical Recurrent Encoder (HRE) [7], Memory Network (MN) [7], Graph Neural Network (GNN) [12], Co-reference Neural Module Network (CorefNMN) [4], Recursive Visual Attention (RVA) [6], Synergistic Network [9], Dual Encoding Visual Dialog”
Learning to Ground Visual Objects for Visual Dialog (2021, methodology): “0 (test-std) MRR ↑ R@1 ↑ R@5 ↑ R@10 ↑ Mean ↓ NDCG ↑ MRR ↑ R@1 ↑ R@5 ↑ R@10 ↑ Mean ↓ LF (Das et al., 2017) 58 els: HCIAE (Lu et al., 2017), CoAtt (Wu et al., 2018), Primary (Guo et al., 2019a), ReDAN (Gan et al., 2019), CorefNMN (Kottur et al., 2018), RvA (Niu et al., 2019), DVAN (Guo et al., 2019b) ”
Modality-Balanced Models for Visual Dialogue (2020, background): “Niu et al. (2018) tries to extract the clues from history recursively while Wu et al. (2018) and Guo, Xu, and Tao (2019) employ co-attention to fuse visual, history, and question features.”
History for Visual Dialog: Do we really need it? (2020, background): “In contrast to previous work on the VisDial dataset, e.g. (Kottur et al., 2018; Agarwal and Goyal, 2018; Gan et al., 2019; Guo et al., 2019; Kang et al., 2019), we are the first to conduct a principled study of dialog history encodings.”
Two Causal Principles for Improving Visual Dialog (2019, methodology): “All of the existing approaches in the VisDial task are based on the typical encoder-decoder framework [15, 12, 33, 11, 32, 48].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “, 2019) VGG LSTM GNN ADAM 7 (Guo et al., 2019) ResNet-101 LSTM Synergistic ADAM 7”
DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue (2019, background): “…(Lu et al. 2016), AMEM (Seo et al. 2017), HCIAE (Lu et al. 2017), SF (Jain, Lazeb-nik, and Schwing 2018), CoAtt (Qi et al. 2018), CorefMN (Kottur et al. 2018), VGNN (Zheng et al. 2019), LF-Att (Das et al. 2017), MN-Att (Das et al. 2017), RvA(Niu et al. 2019) and DL-61(Guo, Xu, and Tao 2019).”