Reasoning with Heterogeneous Graph Alignment for Video Question Answering

By Pin Jiang, Yahong Han. 2020

220 citations recorded by the source corpus.

The dominant video question answering methods are based on fine-grained representation or model-specific attention mechanism. They usually process video and question separately, then feed the representations of different modalities into following late fusion networks. Although these methods use information of one modality to boost the other, they neglect to integrate correlations of both inter- and intra-modality in an uniform module. We propose a deep heterogeneous graph alignment network over the video shots and question words. Furthermore, we explore the network architecture from four steps: representation, fusion, alignment, and reasoning. Within our network, the inter- and intra-modality information can be aligned and interacted simultaneously over the heterogeneous graph and used for cross-modal reasoning. We evaluate our method on three benchmark datasets and conduct extensive ablation study to the effectiveness of the network architecture. Experiments show the network to be superior in quality.

Subfields: Computer Vision and Pattern Recognition, Graph Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Shuicheng Yan
4. Li Fei-Fei
5. Jitendra Malik
6. Xiaogang Wang
7. Trevor Darrell
8. Bernt Schiele

In Graph Learning

1. Shuicheng Yan
2. Angela Yao
3. Tat-seng Chua
4. Byoung-Tak Zhang
5. Pan Zhou
6. Xindi Shang
7. Li Fei-Fei
8. Junbin Xiao

In Question Answering & Reading Comprehension

1. Devshree Patel
2. Ratnam Parikh
3. Yesha Shastri
4. Shuicheng Yan
5. Angela Yao
6. Tat-seng Chua
7. Byoung-Tak Zhang
8. Oswald Lanz

Citations in the corpus

• NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “Using NExT-QA, we evaluate several state-of-the-art (SOTA) video QA techniques [9, 11, 18, 19, 23, 26].”
• Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “This is in contrast to using a classification answer module [33, 36, 37, 42, 105] that can choose only from a fixed predefined vocabulary of answers.”
• Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, background): “QueST (Jiang et al., 2020) Often more significance is given to spatial and temporal features derived from the videos but less to questions, therefore, accounting to this, the authors of [28] propose a novel Question-Guided SpatioTemporal contextual attention network (QueST) method.”
• Video Graph Transformer for Video Question Answering (2022, methodology): “, cross-modal attention [20,33,22], motion-appearance memory [16,14,36], and graph neural networks [23,35,41], have been proposed to model informative videos contents for answering questions.”
• Data augmentation techniques for the Video Question Answering task (2020, background): “based on reasoning techniques which exploit multiple steps LSTM-based neural networks [7] or graphs to better encode the relationships between the visual and textual data [11,13].”
• Revisiting the “Video” in Video-Language Understanding (2022, background): “Standard approaches for addressing these tasks [21,23,29,35,49,55] often operate on a combination of image-derived appearance [7,15] and video-derived motion features [3,36,40,46] as input to an architecture [50, 61] that combines information across the temporal dimension for the final task.”
• DramaQA: Character-Centered Video Story Understanding with Hierarchical QA (2020, methodology): “2019) used the attention mechanism as a reasoning module, and (Jiang and Han 2020) and (Huang et al.”