Video Question Answering via Attribute-Augmented Attention Network Learning

By Yunan Ye, Zhou Zhao, Long Chen, Jun Xiao, Yueting Zhuang. 2017

118 citations recorded by the source corpus.

Video Question Answering is a challenging problem in visual information retrieval, which provides the answer to the referenced video content according to the question. However, the existing visual question answering approaches mainly tackle the problem of static image question, which may be ineffectively for video question answering due to the insufficiency of modeling the temporal dynamics of video contents. In this paper, we study the problem of video question answering by modeling its temporal dynamics with frame-level attention mechanism. We propose the attribute-augmented attention network learning framework that enables the joint frame-level attribute detection and unified video representation learning for video question answering. We then incorporate the multi-step reasoning process for our proposed attention network to further improve the performance. We construct a large-scale video question answering dataset. We conduct the experiments on both multiple-choice and open-ended video question answering tasks to show the effectiveness of the proposed method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “Early VideoQA benchmarks [31, 51, 53, 54, 59, 64] rely on video descriptions [27, 52] (e.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, background): “As result, current VideoQA datasets [15, 19, 32, 40, 43, 63, 69, 76, 81, 82, 83] are limited in size, as the largest, TGIF-QA [32], contains only 72K annotated clips (see Table 1 for more details).”
Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering (2019, methodology): “Many models have been proposed to tackle this problem [5, 10, 27, 30–32].”
Attentive Moment Retrieval in Videos (2018, background): “[46] studied the problem of video question answering by modeling its temporal dynamics with frame-level attention mechanism.”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “Further, [5] studied the problem of modeling temporal dynamics with frame-level attention by proposing an attribute augmented attention network for VQA.”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “[117] Yunan Ye, Zhou Zhao, Yimeng Li, Long Chen, Jun Xiao, and Yueting Zhuang.”
Hierarchical Conditional Relation Networks for Multimodal Video Question Answering (2020, unknown): “Developments along this direction include attribute-based attention [37], hierarchical attention [38,39,40], multi-head attention [41,42], multi-step progressive attention memory [17] or combining self-attention with co-attention [3].”
On Modality Bias in the TVQA Dataset (2020, unknown): “Multiple video question answering (video-QA) datasets have been developed (MovieQA [42], MovieFIB [36], PororoQA [29], TGIF-QA [26], YouTube2Text-QA [47], EgoVQA [15] and TVQA [32]).”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “[60] Yunan Ye, Zhou Zhao, Yimeng Li, Long Chen, Jun Xiao, and Yueting Zhuang.”
Location-Aware Graph Convolutional Networks for Video Question Answering (2020, methodology): “Youtube2Text-QA (Ye et al. 2017) includes the videos from MSVD video set (Chen and Dolan 2011) and the question-answer pairs collected from Youtube2Text (Guadarrama et al. 2013) video description corpus.”
Counterfactual Samples Synthesizing for Robust Visual Question Answering (2020, methodology): “Then both V and Q are fed into the model f vqa to predict answer distributions: Model f vqa typically contains an attention mechanism [13, 29, 41], and it is trained with cross-entropy loss [38, 11].”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, background): “Many video question answering datasets [25, 47, 40, 65, 79] and the methods [80, 49, 36, 73, 76, 20] built on them mainly focus on understanding diverse visual scenes, such as human actions (MovieQA [65]) or 3D object movements without physical and language cues (CATER [25]).”
Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering (2017, background): “2017b), video question answering (Ye et al. 2017) and text-to-image retrieval (Xie, Shen, and Zhu 2016; Li et al.”
EgoVQA - An Egocentric Video Question Answering Benchmark Dataset (2019, methodology): “Instead of simply applying temporal attention once to build the final answer abstraction, several work [21, 54, 60] adopted multi-step reasoning to better relate question words with encoded video features.”
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning (2021, unknown): “While these methods have shown great success in learning to recognize visually complex concepts, such as human activities (Xu et al., 2017; Ye et al., 2017), they typically fail on benchmarks that require the understanding of compositional and causal structures in the videos and text (Yi et al.,…”