Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline

By Vishvak Murahari, Dhruv Batra, Devi Parikh, Abhishek Das. 2019

124 citations recorded by the source corpus.

Prior work in visual dialog has focused on training deep neural models on VisDial in isolation. Instead, we present an approach to leverage pretraining on related vision-language datasets before transferring to visual dialog. We adapt the recently proposed ViLBERT (Lu et al., 2019) model for multi-turn visually-grounded conversations. Our model is pretrained on the Conceptual Captions and Visual Question Answering datasets, and finetuned on VisDial. Our best single model outperforms prior published work (including model ensembles) by more than 1% absolute on NDCG and MRR. Next, we find that additional finetuning using "dense" annotations in VisDial leads to even higher NDCG -- more than 10% over our base model -- but hurts MRR -- more than 17% below our base model! This highlights a trade-off between the two primary metrics -- NDCG and MRR -- which we find is due to dense annotations not correlating well with the original ground-truth answers to questions.

Subfields: Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

1. Zhe Gan
2. Linjie Li
3. Jingjing Liu
4. Devi Parikh
5. Lijuan Wang
6. Yu Cheng
7. Mohit Bansal
8. Steven C. H. Hoi

Citations in the corpus

• Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Multimodal pre-training has brought leaping advances in vision+language understanding tasks such as VQA and VCR, with great potential in extending to visual captioning [85, 71], visual dialog [47, 69], vision-language naviga-”
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, background): “VD-ViLBERT† (Murahari et al., 2020) pre-trains ViLBERT (Lu et al.”
• VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, methodology): “Lastly, we devise a ranking optimization module to further fine-tune on the dense annotations ( § 3.3).”
• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning (2020, background): “They cannot effectively deal with the multi-modal tasks, such as image-text retrieval, image captioning, multimodal machine translation (Lin et al., 2020a; Su et al., 2021) and visual dialog (Murahari et al., 2020).”
• Measuring Social Biases in Grounded Vision and Language Embeddings (2020, methodology): “Grounded embeddings are used for many conse-quential tasks in natural language processing, like visual dialog (Murahari et al., 2019) and visual question answering (Hu et al., 2019).”
• Playing Lottery Tickets with Vision and Language (2021, unknown): “This pretrain-thenfinetune learning paradigm has also been extended to diverse V+L tasks such as novel object captioning [26], visual dialog [51, 71], vision-language navigation [24, 49, 25], OCR-based VQA [73], robust VQA [38], and video-text pre-training [64, 63, 37, 82, 35].”
• VLP: A Survey on Vision-language Pre-training (2022, background): “12-IN-1[115] Image OD-RFs Emb Single-stream No MLM+MVM MTL GQA+GRE+VC+ NLVR+VE+VQA”
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning (2023, methodology): “Following previous works [4, 48, 31], we report the CIDEr score [41] for NoCaps and Flickr30K, iVQA accuracy for iVQA, AUC score for HatefulMemes, and Mean Reciprocal Rank (MRR) for Visual Dialog.”
• Learning to Ground Visual Objects for Visual Dialog (2021, methodology): “VisualBERT (Murahari et al., 2020) and VDBERT (Wang et al.”
• Multi-View Attention Networks for Visual Dialog (2020, methodology): “Moreover, we plan to incorporate the latest pre-training methods (Lu et al., 2019; Murahari et al., 2019) into our model to improve its performance.”
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain (2021, methodology): “In contrast to the boom in generic tasks ( e.g ., VCR [9, 43, 78], VQA [32, 38]), other tasks such as visual relationship detection (RVL-BERT [10]), visual navigation ( i.e ., PERVALENT [24] and VLN-BERT [48]), and visual dialog ( e.g ., VisualD [50], VD-BERT [70]) are still in their infancy.”
• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “10 VisDial-BERT [145] ECCV-2020 image-text Trans MLM, NSP, MIR Pre-training on image-text corpus and finetuning on visual dialog URL”
• Spatially Aware Multimodal Transformers for TextVQA (2020, methodology): “Multimodal representation learning for Vision and Language: Recently, several general architectures for vision and language [10,14,22,24,27,28,33,35, 39, 42, 43, 52] were proposed that reduce architectural differences across tasks.”
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models (2020, methodology): “Pre-trained models [39, 75, 59, 35, 37, 24] such as ViLBERT [43], LXMERT [64] and UNITER [13] have demonstrated great generalizability over diverse V+L tasks [80, 23, 47, 70, 4, 22, 46, 9], such as Visual Question Answering (VQA) [6], Visual Commonsense Reasoning (VCR) [78], and Referring Expression”