Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering

By Pan Lu, Hongsheng Li, Wei Zhang, Jianyong Wang, Xiaogang Wang. 2017

82 citations recorded by the source corpus.

Recently, the Visual Question Answering (VQA) task has gained increasing attention in artificial intelligence. Existing VQA methods mainly adopt the visual attention mechanism to associate the input question with corresponding image regions for effective question answering. The free-form region based and the detection-based visual attention mechanisms are mostly investigated, with the former ones attending free-form image regions and the latter ones attending pre-specified detection-box regions. We argue that the two attention mechanisms are able to provide complementary information and should be effectively integrated to better solve the VQA problem. In this paper, we propose a novel deep neural network for VQA that integrates both attention mechanisms. Our proposed framework effectively fuses features from free-form image regions, detection boxes, and question representations via a multi-modal multiplicative feature embedding scheme to jointly attend question-related free-form image regions and detection boxes for more accurate question answering. The proposed method is extensively evaluated on two publicly available datasets, COCO-QA and VQA, and outperforms state-of-the-art approaches. Source code is available at https://github.com/lupantech/dual-mfa-vqa.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hongsheng Li
2. Xiaogang Wang
3. Jianfeng Gao
4. Mohit Bansal
5. Xiaodan Liang
6. Peng Gao
7. Pan Lu
8. Steven C. H. Hoi

In Computer Vision and Pattern Recognition

1. Xiaogang Wang
2. Luc Van Gool
3. Dacheng Tao
4. Hongsheng Li
5. Ming-Hsuan Yang
6. Yi Yang
7. Wanli Ouyang
8. Trevor Darrell

In Multimodal Learning

1. Hongsheng Li
2. Pan Lu
3. Xiaogang Wang
4. Peng Gao
5. Xiaodan Liang
6. Xiaodong He
7. Zhou Yu
8. Li Deng

In Question Answering & Reading Comprehension

1. Peng Gao
2. Xiaogang Wang
3. Hongsheng Li
4. Zhou Yu
5. M. Zhou
6. Xiaodong He
7. Song-Chun Zhu
8. Jianyong Wang

Citations in the corpus

• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “Inspired by the recent remarkable performance achieved for VQA [32, 31, 9, 8, 25, 7], in this paper, we further extensively benchmark S CIENCE QA with a wide range of attention-based [1, 32, 20, 8] and Transformer-based [29, 25, 26, 7] methods.”
• Visual Question Generation as Dual Task of Visual Question Answering (2017, background): “Recently, two homogenous tasks, Visual Question Answering (VQA) [36, 34, 2, 19] and Visual Question Generation (VQG) [25, 35], have been introduced to the computer vision field as cross-modality learning tasks.”
• R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering (2018, background): “[23] design a novel dual attention network which introduces two types of visual features and enables learning question-releted free-form and detection-based image regions.”
• Question-Guided Hybrid Convolution for Visual Question Answering (2018, methodology): “Similarly, Lu et al [20] employ a co-attention mechanism to simultaneously learn free-form and detection-based image regions related to the input question.”
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, methodology): “from different models, such as ResNet and Faster R-CNN, can be used for multiple image attention mechanisms [157].”
• Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation (2018, background): “…the phrases with a concise subgraph representation, where relationships can be restored with subgraph features (green) and corresponding subject and object. scene graph generation will contribute to the downstream applications such as image retrieval [26,45] and visual question answering [33,38].”
• IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning (2021, background): “Early VQA approaches usually combine multi-modal inputs by applying attention mechanisms over image regions or question words [28, 39, 38, 12, 54, 11].”
• VQA With No Questions-Answers Training (2018, background): “As end-to-end training dominates current answering schemes, many works focused on improving the image-question fused features [10, 8, 45], various attention mechanisms for selecting important features [43, 17, 31, 29, 4] and incorporating outputs of other visual tasks [12, 3, 9, 36].”