DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue

By Xianzhou Jiang, Jiangtao Yu, Zengchang Qin, Yingying Zhuang, Xingxing Zhang, Yue Hu, Qi Wu. 2019

72 citations recorded by the source corpus.

Different from Visual Question Answering task that requires to answer only one question about an image, Visual Dialogue involves multiple questions which cover a broad range of visual content that could be related to any objects, relationships or semantics. The key challenge in Visual Dialogue task is thus to learn a more comprehensive and semantic-rich image representation which may have adaptive attentions on the image for variant questions. In this research, we propose a novel model to depict an image from both visual and semantic perspectives. Specifically, the visual view helps capture the appearance-level information, including objects and their relationships, while the semantic view enables the agent to understand high-level visual semantics from the whole image to the local regions. Futhermore, on top of such multi-view image features, we propose a feature selection framework which is able to adaptively capture question-relevant information hierarchically in fine-grained level. The proposed method achieved state-of-the-art results on benchmark Visual Dialogue datasets. More importantly, we can tell which modality (visual or semantic) has more contribution in answering the current question by visualizing the gate values. It gives us insights in understanding of human cognition in Visual Dialogue.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Devi Parikh
2. Qi Wu
3. Dhruv Batra
4. Dacheng Tao
5. Luc Van Gool
6. Trevor Darrell
7. Chunhua Shen
8. Cordelia Schmid

In Multimodal Learning

1. Devi Parikh
2. Dhruv Batra
3. Qi Wu
4. Abhishek Das
5. Daxin Jiang
6. Xiuyi Chen
7. Can Xu
8. Anton van den Hengel

In Dialogue & Conversational AI

1. Can Xu
2. Daxin Jiang
3. Xiuyi Chen

Citations in the corpus

• Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, methodology): “Recent years have seen incredible progress in Visual Dialog [1, 5–25], spurred in part by the initial efforts of Das et al.”
• Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering (2020, background): “As an extension, [Jiang et al. , 2020] exploits natural language to enrich the graph-based visual representations.”
• Learning to Ground Visual Objects for Visual Dialog (2021, methodology): “GNN-EM (Zheng et al., 2019), FGA (Schwartz et al., 2019), DualVD (Jiang et al., 2020b), CAG (Guo et al., 2020) and KBGN (Jiang et al., 2020a) utilize graph neural networks to obtain the representation of visual objects.”
• Multi-View Attention Networks for Visual Dialog (2020, background): “93 DualVD (Jiang et al. 2020) 56.”
• DialGraph: Sparse Graph Learning Networks for Visual Dialog (2020, unknown): “18 DualVD [18] 56.”