Question-Guided Hybrid Convolution for Visual Question Answering

By Peng Gao, Pan Lu, Hongsheng Li, Shuang Li, Yikang Li, Steven C. H. Hoi, Xiaogang Wang. 2018

71 citations recorded by the source corpus.

In this paper, we propose a novel Question-Guided Hybrid Convolution (QGHC) network for Visual Question Answering (VQA). Most state-of-the-art VQA methods fuse the high-level textual and visual features from the neural network and abandon the visual spatial information when learning multi-modal features. To address these problems, question-guided kernels generated from the input question are designed to convolute with visual features for capturing the textual and visual relationship in the early stage. The question-guided convolution can tightly couple the textual and visual information but also introduce more parameters when learning kernels. We apply the group convolution, which consists of question-independent kernels and question-dependent kernels, to reduce the parameter size and alleviate over-fitting. The hybrid convolution can generate discriminative multi-modal features with fewer parameters. The proposed approach is also complementary to existing bilinear pooling fusion and attention based VQA methods. By integrating with them, our method could further boost the performance. Experiments on VQA datasets validate the effectiveness of QGHC.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “Inspired by the recent remarkable performance achieved for VQA [32, 31, 9, 8, 25, 7], in this paper, we further extensively benchmark S CIENCE QA with a wide range of attention-based [1, 32, 20, 8] and Transformer-based [29, 25, 26, 7] methods.”
Dynamic Neural Networks: A Survey (2021, background): “Cross-Field Image captioning [130], [242], Video captioning [243], [244], Visual question answering [123], [124], [245], Multi-modal sentiment analysis [246], [247]”
Exploring Hate Speech Detection in Multimodal Publications (2019, methodology): “In a recent work, Gao et al. [7] propose a feature fusion scheme to overcome these limitations.”
Language-Conditioned Graph Networks for Relational Reasoning (2019, background): “QGHC [10] predicts questiondependent convolution kernels to modulate visual features.”
Fast Convergence of DETR with Spatially Modulated Co-Attention (2021, methodology): “Motivated by the dynamic filter, QGHC [9] adopted a dynamic group-wise filter to guide the information aggregation in the visual branch using language guided convolution.”
DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue (2019, background): “…context, (Xu and Saenko 2016; Anderson et al. 2018) adopt patches and salient objects learned by attention mechanism as the region context, and (Gao et al. 2018; Li et al. 2019b) exploits inter-object relationships via graph attention networks or convolutional networks to model the relational…”
R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering (2018, methodology): “es perform attention mechanism to improve model capacity. Current attention methods [36, 38] for VQA mainly perform visual attention to learn image regions relevant to the question. Some recent works [8, 9, 13, 14] integrate effective multimodal feature embedding with visual attention to further imp”
PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition (2018, unknown): “In [7], the question-guided hybrid convolution with attention mechanism is proposed to couple the textual and visual information in VQA system.”
Cross Attention Network for Few-shot Classification (2019, methodology): “It has achieved great success in computer vision applications, such as image classification [12, 41, 27], person re-identification [10, 47, 11], image caption [29, 44, 4] and visual question answering [43, 45, 46, 30].”
VQA With No Questions-Answers Training (2018, methodology): “Some methods use dynamic networks with architecture affected by the question [18, 56].”
Character Matters: Video Story Understanding with Character-Aware Relations (2020, unknown): “While some of them (Li et al. 2019; Gao et al. 2018a; Jang et al. 2017) principally focus on factual understanding in short videos, another research direction aims at understanding videos that contain story-lines and answering questions about them.”
Learning Depth-Guided Convolutions for Monocular 3D Object Detection (2019, methodology): “Other techniques like deformable convolution [8] and variants of [20] such as [14, 46, 52, 11], fail to capture object scale and local structure as well.”