Hierarchical Conditional Relation Networks for Video Question Answering

By Thao Minh Le, Vuong Le, Svetha Venkatesh, Truyen Tran. 2020

301 citations recorded by the source corpus.

Video question answering (VideoQA) is challenging as it requires modeling capacity to distill dynamic visual artifacts and distant relations and to associate them with linguistic concepts. We introduce a general-purpose reusable neural unit called Conditional Relation Network (CRN) that serves as a building block to construct more sophisticated structures for representation and reasoning over video. CRN takes as input an array of tensorial objects and a conditioning feature, and computes an array of encoded output objects. Model building becomes a simple exercise of replication, rearrangement and stacking of these reusable units for diverse modalities and contextual information. This design thus supports high-order relational and multi-step reasoning. The resulting architecture for VideoQA is a CRN hierarchy whose branches represent sub-videos or clips, all sharing the same question as the contextual condition. Our evaluations on well-known datasets achieved new SoTA results, demonstrating the impact of building a general-purpose reasoning unit on complex domains such as VideoQA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “HCRN [22] is the recent video question answering model, which involves hierarchical conditional relation networks for better representation relation learning.”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “Using NExT-QA, we evaluate several state-of-the-art (SOTA) video QA techniques [9, 11, 18, 19, 23, 26].”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “More speciﬁcally to the video domain (VideoQA), spatio-temporal video representations in terms of motion and appearance have been used in [23, 27, 33, 35, 36, 37, 42, 43, 44, 50, 87, 89, 97, 105].”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “However, existing works [41, 26, 29, 20, 50, 44] in this field are often evaluated on distinct datasets under different experimental settings, making fair comparison difficult between methods.”
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning (2021, methodology): “Additionally, we also include a recent state-of-the-art VQA model HCRN (Le et al., 2020) for performance comparison, which adopts a conditional relation network for representation and reasoning over videos.”
Experts Collaboration Learning for Continual Multi-Modal Reasoning (2023, methodology): “multi-modal reasoning models [1], [2] that achieve strong performance under the offline learning setting, suffer from catastrophic forgetting when trained under the online continual learning setting.”
Video Graph Transformer for Video Question Answering (2022, methodology): “Then, the output of the ‘[CLS]’ token is fed to a |A|-way classifier in open-ended QA or a 1-way classifier for binary relevance in multichoice QA following [20,28,60].”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, methodology): “We conduct experiments on CLEVRER against several counterparts: TVQA+ [47], Memory [20], IEP (V) [43], TbD-net (V) [57], HCRN [46], MAC [38], NS-DR [77], DCL [16], and Object-based Attention [19].”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “[49] Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran.”
X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval (2022, methodology): “Traditional video-text retrieval methods tend to design task-specific or modality-specific fusion strategies for cross-modal learning from offline extracted video and text features [15, 16, 20, 28, 34, 42, 62], including face recognition/object recognition/audio processing.”
iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, unknown): “While QA based on a single image, i.e., VQA, has been well explored [17, 2, 49, 1, 35, 36], the ﬁeld of VideoQA is now picking up intense momentum [62, 22, 7, 30, 28].”
AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning (2021, methodology): “Using AGQA, we evaluate modern visual reasoning systems (PSAC, HME, and HRCN [11, 33, 36]), and ﬁnd that they barely perform better than models that purely exploit linguistic bias.”