Location-Aware Graph Convolutional Networks for Video Question Answering

By Deng Huang, Peihao Chen, Runhao Zeng, Qing Du, Mingkui Tan, Chuang Gan. 2020

191 citations recorded by the source corpus.

We addressed the challenging task of video question answering, which requires machines to answer questions about videos in a natural language form. Previous state-of-the-art methods attempt to apply spatio-temporal attention mechanism on video frame features without explicitly modeling the location and relations among object interaction occurred in videos. However, the relations between object interaction and their location information are very critical for both action recognition and question reasoning. In this work, we propose to represent the contents in the video as a location-aware graph by incorporating the location information of an object into the graph construction. Here, each node is associated with an object represented by its appearance and location features. Based on the constructed graph, we propose to use graph convolution to infer both the category and temporal locations of an action. As the graph is built on objects, our method is able to focus on the foreground action contents for better video question answering. Lastly, we leverage an attention mechanism to combine the output of graph convolution and encoded question features for final answer reasoning. Extensive experiments demonstrate the effectiveness of the proposed methods. Specifically, our method significantly outperforms state-of-the-art methods on TGIF-QA, Youtube2Text-QA and MSVD-QA datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Graph Learning

Citations in the corpus

NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “During aggregation, the textual clues from the question side (usually modeled by RNNs) are integrated for languageguided video reasoning and are achieved by additional reasoning modules, such as spatial and temporal attention [17, 18, 20, 60], co-attention [19, 26, 30], multi-cycle memory [9, 11], g”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “This is in contrast to using a classiﬁcation answer module [33, 36, 37, 42, 105] that can choose only from a ﬁxed predeﬁned vocabulary of answers.”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “9% with one of the latest models L-GCN [21].”
Dense Regression Network for Video Grounding (2020, background): “This task has drawn increasing attention over the past few years due to its vast potential applications in video understanding [38, 3, 45, 44, 6], video retrieval [42, 8], and human-computer interaction [35, 20, 50], etc.”
Video Graph Transformer for Video Question Answering (2022, background): “However, these graph methods either construct monolithic graphs that do not disambiguate between relations in 1) space and time, 2) local and global scopes [19,57], or build static graphs at frame-level without explicitly capturing the temporal dynamics [36,42,60].”
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning (2021, unknown): “Later, visual reasoning was extended to more complex dynamic videos (Lei et al., 2018; Fan et al., 2019; Li et al., 2020; 2019a; Huang et al., 2020).”
RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning (2020, background): “Video analysis [24] has been a prominent research topic in computer vision due to its vast potential applications, such as action recognition [32, 8], event detection [14], action localization [4, 49, 50], audio-visual scene analysis [16, 5, 12], etc .”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “A vast majority of VideoQA approaches rely on relatively small, manually annotated VideoQA datasets [3, 10, 11, 16–18, 23, 26, 27, 33, 36, 37, 40–43, 49, 50, 53, 64, 67, 76, 77, 83, 88, 89, 93, 100, 108, 112, 115, 117, 124, 128].”
Data augmentation techniques for the Video Question Answering task (2020, background): “based on reasoning techniques which exploit multiple steps LSTM-based neural networks [7] or graphs to better encode the relationships between the visual and textual data [11,13].”
Cascade Reasoning Network for Text-based Visual Question Answering (2020, methodology): “[20] propose to model object realtionship using graph convolutional networks.”
Towards Accurate Text-based Image Captioning with Content Diversity Exploration (2021, background): “In the image captioning area, the texts contained in images are also of critical importance and often provide valuable information [5, 19, 20, 34, 41] for caption generation.”
DramaQA: Character-Centered Video Story Understanding with Hierarchical QA (2020, methodology): “2019) used the attention mechanism as a reasoning module, and (Jiang and Han 2020) and (Huang et al. 2020) utilized graphs for modeling high-level temporal structure.”
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, background): “Based on techniques of speech and image synthesis, the study in [28] summarizes existing works on the generation of gestures with simultaneous speeches based on deep generative models.”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, background): “Many video question answering datasets [25, 47, 40, 65, 79] and the methods [80, 49, 36, 73, 76, 20] built on them mainly focus on understanding diverse visual scenes, such as human actions (MovieQA [65]) or 3D object movements without physical and language cues (CATER [25]).”