Open-Ended Long-form Video Question Answering via Adaptive Hierarchical Reinforced Networks

By Zhou Zhao, Zhu Zhang, Shuwen Xiao, Zhou Yu, Jun Yu, Deng Cai, Fei Wu, Yueting Zhuang. 2018

61 citations recorded by the source corpus.

Open-ended long-form video question answering is challenging problem in visual information retrieval, which automatically generates the natural language answer from the referenced long-form video content according to the question. However, the existing video question answering works mainly focus on the short-form video question answering, due to the lack of modeling the semantic representation of long-form video contents. In this paper, we consider the problem of long-form video question answering from the viewpoint of adaptive hierarchical reinforced encoder-decoder network learning. We propose the adaptive hierarchical encoder network to learn the joint representation of the long-form video contents according to the question with adaptive video segmentation. we then develop the reinforced decoder network to generate the natural language answer for open-ended video question answering. We construct a large-scale long-form video question answering dataset. The extensive experiments show the effectiveness of our method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tat-seng Chua
2. Devshree Patel
3. Ratnam Parikh
4. Yesha Shastri
5. Yueting Zhuang
6. Zhou Yu
7. Dacheng Tao
8. Zhou Zhao

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Qi Tian
4. Tat-seng Chua
5. Yi Yang
6. Cordelia Schmid
7. Xiaogang Wang
8. Jun Yu

In Question Answering & Reading Comprehension

1. Devshree Patel
2. Ratnam Parikh
3. Yesha Shastri
4. Tat-seng Chua
5. Junbin Xiao
6. Jun Xiao
7. Xindi Shang
8. Angela Yao

Citations in the corpus

• NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “It can be regarded as a soft version of accuracy that factors in synonyms and other semantics [53, 61]”
• Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “To address this issue, several works have explored leveraging manually annotated video descriptions [35, 82, 87, 96, 98, 99, 100] for automatic generation of VideoQA datasets, using rule-based [30, 66] approaches.”
• Deep Modular Co-Attention Networks for Visual Question Answering (2019, background): “Significant progress has been made in many vision-language tasks, including image-text matching [23, 14], visual captioning [9, 30, 1], visual grounding [10, 34] and visual question answering (VQA) [2, 21, 14, 36].”
• Multi-interaction Network with Object Relation for Video Question Answering (2019, methodology): “Zhao et al. [42] try to process longer video by using an adaptive hierarchical encoder which could learn segment-level video representation with adaptive video segmentation.”
• Spatiotemporal-Textual Co-Attention Network for Video Question Answering (2019, methodology): “VQA has attracted extensive attention in recent years [9, 29, 59], but it is still a challenging task relying on the accurate understanding of visual content and questions as well as the relevance between them [18, 33].”
• Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “A new large scale long-form VQA dataset, created from the ActivityNet data [50], is also contributed by [11].”