Iterative Context-Aware Graph Inference for Visual Dialog

By Dan Guo, Haibo Wang, Hanwang Zhang, Zhengjun Zha, Meng Wang. 2020

54 citations recorded by the source corpus.

Visual dialog is a challenging task that requires the comprehension of the semantic dependencies among implicit visual and textual contexts. This task can refer to the relation inference in a graphical model with sparse contexts and unknown graph structure (relation descriptor), and how to model the underlying context-aware relation inference is critical. To this end, we propose a novel Context-Aware Graph (CAG) neural network. Each node in the graph corresponds to a joint semantic feature, including both object-based (visual) and history-related (textual) context representations. The graph structure (relations in dialog) is iteratively updated using an adaptive top-K message passing mechanism. Specifically, in every message passing step, each node selects the most K relevant nodes, and only receives messages from them. Then, after the update, we impose graph attention on all the nodes to get the final graph embedding and infer the answer. In CAG, each node has dynamic relations in the graph (different related K neighbor nodes), and only the most relevant nodes are attributive to the context-aware relational graph inference. Experimental results on VisDial v0.9 and v1.0 datasets show that CAG outperforms comparative methods. Visualization results further validate the interpretability of our method.

Subfields: Computer Vision and Pattern Recognition, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Hueiseok Lim
2. Xiuyi Chen
3. Can Xu
4. Daxin Jiang
5. Sungjin Park
6. Y. Yoon
7. Taesun Whang

In Dialogue & Conversational AI

1. Xiuyi Chen
2. Can Xu
3. Daxin Jiang

Citations in the corpus

• Learning to Ground Visual Objects for Visual Dialog (2021, methodology): “GNN-EM (Zheng et al., 2019), FGA (Schwartz et al., 2019), DualVD (Jiang et al., 2020b), CAG (Guo et al., 2020) and KBGN (Jiang et al., 2020a) utilize graph neural networks to obtain the representation of visual objects.”
• Multi-View Attention Networks for Visual Dialog (2020, background): “Another direction of research that is inspired by graphical networks focuses on learning the inherent relations among image, dialog history, and question (Zheng et al. 2019; Schwartz et al. 2019; Guo et al. 2020).”