Explore Multi-Step Reasoning in Video Question Answering

By Yahong Han. 2018

58 citations recorded by the source corpus.

This invited talk is a repeated but more detailed talk about the paper which is accepted by ACM-MM 2018: Video question answering (VideoQA) always involves visual reasoning. When answering questions composing of multiple logic correlations, models need to perform multi-step reasoning. In this paper, we formulate multi-step reasoning in VideoQA as a new task to answer compositional and logical structured questions based on video content. Existing VideoQA datasets are inadequate as benchmarks for the multi-step reasoning due to limitations as lacking logical structure and having language biases. Thus we design a system to automatically generate a large-scale dataset, namely SVQA (Synthetic Video Question Answering). Compared with other VideoQA datasets, SVQA contains exclusively long and structured questions with various spatial and temporal relations between objects. More importantly, questions in SVQA can be decomposed into human readable logical tree or chain layouts, each node of which represents a sub-task requiring a reasoning operation such as comparison or arithmetic. Towards automatic question answering in SVQA, we develop a new VideoQA model. Particularly, we construct a new attention module, which contains spatial attention mechanism to address crucial and multiple logical sub-tasks embedded in questions, as well as a refined GRU called ta-GRU (temporal-attention GRU) to capture the long-term temporal dependency and gather complete visual cues. Experimental results show the capability of multi-step reasoning of SVQA and the effectiveness of our model when compared with other existing models.

Subfields: Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “As result, current VideoQA datasets [12, 17, 18, 22, 28, 35, 40, 45, 48, 62, 71, 77, 87, 93, 94, 95, 96] are limited in size, as the largest, TGIF-QA [35], contains only 72K annotated clips (see Figure 2 for more details).”
CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning (2019, methodology): “…cls 44 TFGIF-QA (Jang et al., 2017) 72K 3s qa - MovieQA (Tapaswi et al., 2016) 400 200s qa - Robot Pushing (Finn et al., 2016) 57K 1s gen - SVQA (Song et al., 2018) 12K 4s qa - Moving MNIST (Srivastava et al., 2015) - 2s gen - Flash MNIST (Long et al., 2018) spatiotemporal deep networks…”
Reinforcement Learning with Attention that Works: A Self-Supervised Approach (2019, methodology): “Attention models were applied with remarkable success to complex visual tasks such as video and scene understanding [10,20,6], natural language understanding including machine translation [1,30], and generative models using generative adversarial networks (GANs) [27,13].”
Multi-interaction Network with Object Relation for Video Question Answering (2019, background): “[28] formulate multi-step reasoning in video QA as a new task to answer compositional and logical structured questions based on video content.”
Hierarchical Conditional Relation Networks for Multimodal Video Question Answering (2020, methodology): “Multi-step reasoning for Video QA is also approached by Xu et al. (2017) and Song et al. (2018) with refined attention.”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, background): “[93] Xiaomeng Song, Yucheng Shi, Xin Chen, and Yahong Han.”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “[44] Xiaomeng Song, Yucheng Shi, Xin Chen, and Yahong Han.”
Neural Reasoning, Fast and Slow, for Video Question Answering (2019, methodology): “compositional linguistic cues. Although it is tempting to extrapolate from the recent successes on visual QA over static images [1, 10, 33, 35], Video QA is relatively new and great challenges remain [31,22]. While acquiring visual knowledge of objects and relations from static images has advanced h”