Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
By Zhe Gan, Yu Cheng, Ahmed El Kholy, Linjie Li, Jingjing Liu, Jianfeng Gao. 2019
111 citations recorded by the source corpus.
This paper presents a new model for visual dialog, Recurrent Dual Attention Network (ReDAN), using multi-step reasoning to answer a series of questions about an image. In each question-answering turn of a dialog, ReDAN infers the answer progressively through multiple reasoning steps. In each step of the reasoning process, the semantic representation of the question is updated based on the image and the previous dialog history, and the recurrently-refined representation is used for further reasoning in the subsequent step. On the VisDial v1.0 dataset, the proposed ReDAN model achieves a new state-of-the-art of 64.47% NDCG score. Visualization on the reasoning process further demonstrates that ReDAN can locate context-relevant visual and textual clues via iterative refinement, which can lead to the correct answer step-by-step.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Zhe Gan
- Luc Van Gool
- Cordelia Schmid
- Trevor Darrell
- Mohit Bansal
- Xiaogang Wang
- Steven C. H. Hoi
In Multimodal Learning
In Dialogue & Conversational AI
- Takayuki Okatani
- Van-Quang Nguyen
- Masanori Suganuma
- Hanwang Zhang
- Zhengjun Zha
- Xiuyi Chen
- Can Xu
- Daxin Jiang
Citations in the corpus
- Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm (2021, background): “This problem has been recognized in previous work which has sought to allow serial reasoning using specialized neural network architectures [7, 26].”
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Prominent work includes: (i) advanced attention mechanisms [40, 79, 51]; (ii) better multimodal fusion methods [14, 80, 27, 26]; (iii) multi-step reasoning [77, 22, 15]; (iv) incorporation of object relations [55, 48, 6, 32]; and (v) neural module networks for compositional reasoning [3, 25, 20, 11]”
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, background): “We follow the discriminative setting where the model ranks a pool of answer candidates (Gan et al., 2019; Wang et al., 2020; Murahari et al., 2020).”
- VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, methodology): “First, we employ a unified vision-dialog Trans-former to encode both the image and dialog history, where we append an answer candidate ˆ A t in the input to model their interactions in an early fusion manner ( § 3.1).”
- MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation (2022, background): “Several modeling works (Qi et al., 2020; Niu et al., 2019; Gan et al., 2019) investigate how to escalate the performance of conversational agents in image-grounded dialogue.”
- Two Causal Principles for Improving Visual Dialog (2019, methodology): “Thanks to the recent advances in encoder-decoder frameworks in VQA [22, 38] and natural language process-*Equal contribution ing [39], the performance (NDCG [1]) of VisDial in literature is significantly improved from the baseline 51.63% [2] to the state-of-the-art 64.47% [11].”
- DialGraph: Sparse Graph Learning Networks for Visual Dialog (2020, methodology): “…on visual dialog into three groups: (1) soft attention-based methods that compute attended representations of the image, and the history [8,28,42,13,9,38,30], (2) a visual coreference resolution [39,25,32,19] that clarifies ambiguous expressions ( e.g ., it, them) in the question and links them to…”
- Efficient Attention Mechanism for Handling All the Interactions between Many Inputs with Application to Visual Dialog (2019, methodology): “To be specific, we use the ac-Table curacy values reported in [13] for a fair comparison, in which these methods are reimplemented using the bottom-up-attention features.”
- Learning to Ground Visual Objects for Visual Dialog (2021, methodology): “ReDAN (Gan et al., 2019) and DMAM (Chen et al., 2020a) use multi-step reasoning based on dual attention to iteratively update related visual objects.”
- Iterative Context-Aware Graph Inference for Visual Dialog (2020, methodology): “To improve performance, various attention mechanisms have been widely used in the task, including history-conditioned image attention (HCIAE) [23], sequential co-attention (CoAtt) [34], dual visual attention (DVAN) [9], and recurrent dual attention (ReDAN) [7].”
- History for Visual Dialog: Do we really need it? (2020, background): “In contrast to previous work on the VisDial dataset, e.g. (Kottur et al., 2018; Agar-wal and Goyal, 2018; Gan et al., 2019; Guo et al., 2019; Kang et al., 2019), we are the first to conduct a principled study of dialog history encodings.”
- Violin: A Large-Scale Dataset for Video-and-Language Inference (2020, unknown): “Many neural network models have been proposed for these tasks, such as more advanced attention mechanisms [64, 43, 70], better multimodal fusion methods [15, 71, 31, 30], the use of multi-step reasoning [24, 17, 7], the incorporation of relations [49, 38, 45], and neural module networks for composit”
- Modality-Balanced Models for Visual Dialogue (2020, result): “Although ReDAN+ (Gan et al. 2019) and MReaL– BDAI show higher NDCG scores, our consensus dropout fusion model shows more balanced results over metrics while still having a competitive NDCG score compared to DAN (Kang, Lim, and Zhang 2019), with rank 3 based on NDCG metric and high balance rank…”
- Recent advances in deep learning based dialogue systems: a systematic survey (2021, methodology): “Visual dialogue can be seen as a multi-step reasoning process over a series of questions (Gan et al., 2019). Gan et al. (2019) learned semantic representation of the question based on dialogue history and a given image, and recurrently updated the representation. Shuster et al. (2019) proposed a set”
- Multi-View Attention Networks for Visual Dialog (2020, methodology): “Our proposed approach performs better with respect to all metrics than ReDAN (Gan et al. 2019), which averages the ranking results of the discriminative and generative model, and LTMI (Nguyen, Suganuma, and Okatani 2020), which employs multi-task learning but uses only discriminative decoder outputs”