Visual Coreference Resolution in Visual Dialog using Neural Module Networks

By Satwik Kottur, José M. F. Moura, Devi Parikh, Dhruv Batra, Marcus Rohrbach. 2018

174 citations recorded by the source corpus.

Visual dialog entails answering a series of questions grounded in an image, using dialog history as context. In addition to the challenges found in visual question answering (VQA), which can be seen as one-round dialog, visual dialog encompasses several more. We focus on one such problem called visual coreference resolution that involves determining which words, typically noun phrases and pronouns, co-refer to the same entity/object instance in an image. This is crucial, especially for pronouns (e.g., ‘it’), as the dialog agent must first link it to a previous coreference (e.g., ‘boat’), and only then can rely on the visual grounding of the coreference ‘boat’ to reason about the pronoun ‘it’. Prior work (in visual dialog) models visual coreference resolution either (a) implicitly via a memory network over history, or (b) at a coarse level for the entire question; and not explicitly at a phrase level of granularity. In this work, we propose a neural module network architecture for visual dialog by introducing two novel modules—Refer and Exclude—that perform explicit, grounded, coreference resolution at a finer word level. We demonstrate the effectiveness of our model on MNIST Dialog, a visually simple yet coreference-wise complex dataset, by achieving near perfect accuracy, and on VisDial, a large and challenging visual dialog dataset on real images, where our model outperforms other approaches, and is more interpretable, grounded, and consistent qualitatively.

Subfields: Computer Vision and Pattern Recognition, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Extraction

Citations in the corpus

Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, background): “Follow-up works on visual dialog have explored the use of deep reinforcement learning [3, 4, 17], knowledge transfer from discriminative to generative decoders [5], conditional variational autoencoders [6], generative adversarial networks [7], attention mechanisms for visual coreference resolution [”
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, methodology): “ReDAN outperforms the published state-of-the-art method, CorefNMN [26], by a signiﬁcant margin, lifting NDCG from 54.70 to 57.63, and MRR from 61.”
Look Before you Speak: Visually Contextualized Utterances (2020, background): “Popular vision and language tasks include visual question answering [4, 5, 22, 28, 58, 61], visual dialog [15, 16, 36, 40, 67], visual captioning [56, 59, 68, 73, 86], visual grounding [18, 22, 57, 83] and video-text retrieval [7, 25, 49, 53, 63].”
Dual Attention Networks for Visual Reference Resolution in Visual Dialog (2019, methodology): “…HRE [Das et al. , 2017]), (2) Attention-based approaches (MN [Das et al. , 2017], HCIAE [Lu et al. , 2017] and CoAtt [Wu et al. , 2018]), and (3) Approaches that deal with visual reference resolution in VisDial (AMEM [Seo et al. , 2017], CorefNMN [Kottur et al. , 2018] and RvA [Niu et al. , 2018]).”
Two Causal Principles for Improving Visual Dialog (2019, background): “1) Holistic: they treat history as a whole to feed into models like HACAN [43], DAN [16] and CorefNMN [18].”
History for Visual Dialog: Do we really need it? (2020, background): “In contrast to previous work on the VisDial dataset, e.g. (Kottur et al., 2018; Agar-wal and Goyal, 2018; Gan et al., 2019; Guo et al., 2019; Kang et al., 2019), we are the ﬁrst to conduct a principled study of dialog history encodings.”
FindIt: Generalized Localization with Natural Language Queries (2022, background): “Large amounts of work have been done to learn across image and language, including visualgrounding [16,81,49,96,91,62], image captioning [6,1,5], visual question and answering (VQA) [33,12], visual dialog [11,39], visual reasoning [75,93,85], text-toimage and image-to-text retrieval [32,66,74], and ”
Learning to Ground Visual Objects for Visual Dialog (2021, methodology): “MN (Das et al., 2017), HCIAE (Lu et al., 2017), CorefNMN (Kottur et al., 2018), CoAtt (Wu et al., 2018), RvA (Niu et al., 2019), DVAN (Guo et al., 2019b) utilize kinds of attention mechanisms as the backbone to locate the related visual objects.”
Efficient Attention Mechanism for Handling All the Interactions between Many Inputs with Application to Visual Dialog (2019, methodology): “…all the interactions over many utilities via a factor graph; iii) methods that attempt to resolve visual co-reference, e.g., RvA [30] and CorefNMN [22], which use neural modules to form an attention mechanism, DAN [19], which employs a network having two attention modules, and AMEM [35], which…”
Grounded Agreement Games: Emphasizing Conversational Grounding in Visual Dialogue Settings (2019, background): “Later work by Kottur et al. (2018) on the “visual dialogue” dataset (Das et al., 2017) indeed identified co-reference in the questions as the main issue that distinguishes this setting from one-shot question answering.”
Factor Graph Attention (2019, background): “1, this results in a signiﬁcant improvement of performance, even when compared to the very recently published baselines [21, 53, 27].”
VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs (2021, background): “Several approaches have been introduced for these tasks [34, 3, 23, 42, 16, 41, 25, 52, 20, 27, 30, 55, 17, 31].”
DialGraph: Sparse Graph Learning Networks for Visual Dialog (2020, methodology): “…methods that compute attended representations of the image, and the history [8,28,42,13,9,38,30], (2) a visual coreference resolution [39,25,32,19] that clariﬁes ambiguous expressions ( e.g ., it, them) in the question and links them to a speciﬁc entity in the image, and (3) a…”
Visually Grounded Neural Syntax Acquisition (2019, unknown): “Finally, it may be possible to extend our approach to other linguistic tasks such as dependency parsing (Christie et al., 2016b), coreference resolution (Kottur et al., 2018), and learning pragmatics beyond semantics (Andreas and Klein, 2016).”
Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems (2019, methodology): “CorefNMN (Kottur et al., 2018) 54.”
Making History Matter: Gold-Critic Sequence Training for Visual Dialog (2019, background): “A good dialog agent should accomplish a complex series of reasoning sub-tasks: contextual visual perception [31, 21, 6, 28, 18, 32, 29], language modeling [15, 5], and co-reference resolution in dia-Figure log history [28, 18] ( e.g ., identify what is “that”).”
Large-Scale Answerer in Questioner's Mind for Visual Dialog Question Generation (2019, methodology): “The performance of AQM+ can be boosted further by employing the models recently proposed in the visual dialog field such as other question generator models [11] and question answering models [17].”
Reasoning Visual Dialogs With Structural and Partial Observations (2019, methodology): “More recently, [28] used a neural module network to solve the problem of visual coreference resolution.”