Reasoning Visual Dialogs With Structural and Partial Observations

By Zilong Zheng, Wenguan Wang, Siyuan Qi, Song-Chun Zhu. 2019

121 citations recorded by the source corpus.

We propose a novel model to address the task of Visual Dialog which exhibits complex dialog structures. To obtain a reasonable answer based on the current question and the dialog history, the underlying semantic dependencies between dialog entities are essential. In this paper, we explicitly formalize this task as inference in a graphical model with partially observed nodes and unknown graph structures (relations in dialog). The given dialog entities are viewed as the observed nodes. The answer to a given question is represented by a node with missing value. We first introduce an Expectation Maximization algorithm to infer both the underlying dialog structures and the missing node values (desired answers). Based on this, we proceed to propose a differentiable graph neural network (GNN) solution that approximates this process. Experiment results on the VisDial and VisDial-Q datasets show that our model outperforms comparative methods. It is also observed that our method can infer the underlying dialog structure for better dialog reasoning.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Dacheng Tao
3. Linjie Li
4. Jianfeng Gao
5. Hanwang Zhang
6. Jingjing Liu
7. Luc Van Gool
8. Yu Cheng

In Multimodal Learning

1. Zhe Gan
2. Linjie Li
3. Jingjing Liu
4. Jianfeng Gao
5. Yu Cheng
6. Hanwang Zhang
7. Lijuan Wang
8. Takayuki Okatani

Citations in the corpus

• VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, neutral): “Discriminative Setting Generative Setting MRR ↑ R@1 ↑ R@5 ↑ R@10 ↑ Mean ↓ MRR ↑ R@1 ↑ R@5 ↑ R@10 ↑ Mean ↓ LF (Das et HACAN (Yang et al., 2019), Synergistic (Guo et al., 2019a), DAN (Kang et al., 2019), and ReDAN (Gan et al., 2019).”
• Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, neutral): “P ub lis he d R es ul ts  GNN [12] 52.”
• Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, background): “, 2019), graph neural network (Zheng et al., 2019), history-advantage sequence training (Yang et al.”
• Efficient Attention Mechanism for Handling All the Interactions between Many Inputs with Application to Visual Dialog (2019, methodology): “There are also other approaches that are not fit for the above categorizations, including GNN [48], which models relations in a dialog by an unknown graph structure to resolve via an EM algorithm and [4, 9], which use reinforcement learning to train two agents to play a guessing game.”
• A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “has been used to generate new knowledge about vision and language [187]–[192].”
• Confidence-aware Non-repetitive Multimodal Transformers for TextCaps (2020, unknown): “In recent years, many works have focused on vision or language tasks (Zheng et al. 2019; Gao et al. 2020; Liao et al. 2020).”
• DialGraph: Sparse Graph Learning Networks for Visual Dialog (2020, background): “Zheng et al. [47] designed a structure inference model while predicting the answer in the context of an expectation-maximization (EM) algorithm.”
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “, 2019) ResNet-152 LSTM RvA ADAM 7 (Zheng et al., 2019) VGG LSTM GNN ADAM 7 (Guo et al.”
• Vision-Dialog Navigation by Exploring Cross-Modal Memory (2020, methodology): “Our work is related to some visual dialogue methods [8, 19, 25, 29, 12, 37, 16] for visual coreference resolution.”
• Learning to Ground Visual Objects for Visual Dialog (2021, methodology): “GNN-EM (Zheng et al., 2019), FGA (Schwartz et al., 2019), DualVD (Jiang et al., 2020b), CAG (Guo et al., 2020) and KBGN (Jiang et al., 2020a) utilize graph neural networks to obtain the representation of visual objects.”
• Two Causal Principles for Improving Visual Dialog (2019, methodology): “All of the existing approaches in the VisDial task are based on the typical encoder-decoder framework [15, 12, 33, 11, 32, 48].”
• Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks (2019, unknown): “Recently, GNNs have been successfully applied in many fields, including molecular biology [14], computer vision [48, 71, 76], machine learning [62] and natural language processing [2].”
• Multi-View Attention Networks for Visual Dialog (2020, background): “Another direction of research that is inspired by graphical networks focuses on learning the inherent relations among image, dialog history, and question (Zheng et al. 2019; Schwartz et al. 2019; Guo et al. 2020).”