Modality-Balanced Models for Visual Dialogue

By Hyounghun Kim, Hao Hao Tan, Mohit Bansal. 2020

29 citations recorded by the source corpus.

The Visual Dialog task requires a model to exploit both image and conversational context information to generate the next response to the dialogue. However, via manual analysis, we find that a large number of conversational questions can be answered by only looking at the image without any access to the context history, while others still need the conversation context to predict the correct answers. We demonstrate that due to this reason, previous joint-modality (history and image) models over-rely on and are more prone to memorizing the dialogue history (e.g., by extracting certain keywords or patterns in the context information), whereas image-only models are more generalizable (because they cannot memorize or extract keywords from history) and perform substantially better at the primary normalized discounted cumulative gain (NDCG) task metric which allows multiple correct answers. Hence, this observation encourages us to explicitly maintain two models, i.e., an image-only model and an image-history joint model, and combine their complementary abilities for a more balanced multimodal model. We present multiple methods for this integration of the two models, via ensemble and consensus dropout fusion with shared parameters. Empirically, our models achieve strong results on the Visual Dialog challenge 2019 (rank 3 on NDCG and high balance across metrics), and substantially outperform the winner of the Visual Dialog challenge 2018 on most metrics.

Subfields: Computer Vision and Pattern Recognition, Language and Linguistics, Multimodal Learning, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Language and Linguistics

In Multimodal Learning

In Dialogue & Conversational AI

Caiming Xiong

Citations in the corpus

VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, methodology): “Sur-prisingly, the model initialized with the weights from VLP that was pretrained on Conceptual Cap-tions (Sharma et al., 2018), does not work better than the one initialized from BERT.”
DialGraph: Sparse Graph Learning Networks for Visual Dialog (2020, background): “The CDF [20] randomly extracted up to three elements of the dialog history to avoid excessive exploitation of the whole history.”
Multi-View Attention Networks for Visual Dialog (2020, background): “Recent approaches (Murahari et al., 2019; Kim et al., 2020; Nguyen et al., 2019) reported that they observed a trade-off relationship between two primary metrics (i.”