Multi-Modality Latent Interaction Network for Visual Question Answering

By Peng Gao, Haoxuan You, Zhanpeng Zhang, Xiaogang Wang, Hongsheng Li. 2019

87 citations recorded by the source corpus.

Exploiting relationships between visual regions and question words have achieved great success in learning multi-modality features for Visual Question Answering (VQA). However, we argue that existing methods mostly model relations between individual visual regions and words, which are not enough to correctly answer the question. From humans' perspective, answering a visual question requires understanding the summarizations of visual and language information. In this paper, we proposed the Multi-modality Latent Interaction module (MLI) to tackle this problem. The proposed module learns the cross-modality relationships between latent visual and language summarizations, which summarize visual regions and question into a small number of latent representations to avoid modeling uninformative individual region-word relations. The cross-modality information between the latent summarizations are propagated to fuse valuable information from both modalities and are used to update the visual and word features. Such MLI modules can be stacked for several stages to model complex and latent relations between the two modalities and achieves highly competitive performance on public VQA benchmarks, VQA v2.0 and TDIUC . In addition, we show that the performance of our methods could be significantly improved by combining with pre-trained language model BERT.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Qi Tian
2. Yi Yang
3. Xiaogang Wang
4. Hongsheng Li
5. Dacheng Tao
6. Ling Shao
7. Luc Van Gool
8. Liang Zheng

In Multimodal Learning

1. Peng Gao
2. Hongsheng Li
3. Jifeng Dai
4. Xiaogang Wang
5. Qi Tian
6. Chi Su
7. Rongrong Ji
8. Qingming Huang

In Question Answering & Reading Comprehension

1. Peng Gao
2. Xiaogang Wang
3. Qi Tian
4. Hongsheng Li

Citations in the corpus

• Interpretable Visual Reasoning via Probabilistic Formulation Under Natural Supervision (2020, methodology): “A widely-used approach is to perform reasoning by sequential interactions between image representations and question embeddings [57,23,31,14,15].”
• Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks (2022, background): “have been proposed to achieve the state-of-the-art performance in VQA [2], [6], [7], [48] and image captioning [3], [9], [10], [49]–[53].”
• Character Matters: Video Story Understanding with Character-Aware Relations (2020, unknown): “…in recent works that scene graphs can provide rich knowledge of image semantics and help boost high-level tasks such as Image Cap-tioning and Visual Question Answering (Yao et al. 2018; Liang et al. 2019; Yang et al. 2018; Gao et al. 2019a; 2018b; 2019b; Geng et al. 2019; Shi et al. 2020).”
• Fast Convergence of DETR with Spatially Modulated Co-Attention (2021, methodology): “Transformer has achieved success on machine translation, after which Transformer has been adopted in different fields, including model pre-training [6, 29, 30, 3], visual recognition [28, 7], and multi-modality fusion [40, 8, 27, 10, 11].”