Joint Video and Text Parsing for Understanding Events and Answering Queries

By Kewei Tu, Meng Meng, Mun Wai Lee, Tae Eun Choe, Song-Chun Zhu. 2013

121 citations recorded by the source corpus.

This article proposes a multimedia analysis framework to process video and text jointly for understanding events and answering user queries. The framework produces a parse graph that represents the compositional structures of spatial information (objects and scenes), temporal information (actions and events), and causal information (causalities between events and fluents) in the video and text. The knowledge representation of the framework is based on a spatial-temporal-causal AND-OR graph (S/T/C-AOG), which jointly models possible hierarchical compositions of objects, scenes, and events as well as their interactions and mutual contexts, and specifies the prior probabilistic distribution of the parse graphs. The authors present a probabilistic generative model for joint parsing that captures the relations between the input video/text, their corresponding parse graphs, and the joint parse graph. Based on the probabilistic model, the authors propose a joint parsing system consisting of three modules: video parsing, text parsing, and joint inference. Video parsing and text parsing produce two parse graphs from the input video and text, respectively. The joint inference module produces a joint parse graph by performing matching, deduction, and revision on the video and text parse graphs. The proposed framework has the following objectives: to provide deep semantic parsing of video and text that goes beyond the traditional bag-of-words approaches; to perform parsing and reasoning across the spatial, temporal, and causal dimensions based on the joint S/T/C-AOG representation; and to show that deep joint parsing facilitates subsequent applications such as generating narrative text descriptions and answering queries in the forms of who, what, when, where, and why. The authors empirically evaluated the system based on comparison against ground-truth as well as accuracy of query answering and obtained satisfactory results.

Subfields: Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “These approaches have been extended to the video domain as well [20, 34, 42].”
Visual Dialog (2016, background): “tion [55], object detection [27] – to ‘high-level’ AI tasks such as learning to play Atari video games [35] and Go [47], answering reading comprehension questions by understanding short stories [15, 57], and even answering questions about images [4, 32, 41] and videos [49, 50]!”
VQA: Visual Question Answering (2015, background): “The proposed VQA task has connections to other related work: [42] has studied joint parsing of videos and corresponding text to answer queries on two datasets containing 15 video clips each.”
Visual7W: Grounded Question Answering in Images (2015, background): “The proposed models range from probabilistic inference [27, 44, 51] and recurrent neural networks [1, 7, 28, 36] to convolutional networks [26].”
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question (2015, background): “There has been recent effort on the visual question answering task [9, 2, 22, 37].”
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering (2015, background): “It has many real-life applications, such as automatic querying of surveillance video [4] or assisting the visually impaired [5].”
Yin and Yang: Balancing and Answering Binary Visual Questions (2015, background): “Recent work has proposed several datasets and methods to promote research on the task of visual question answering [19, 4, 37, 28, 2, 29, 18, 32], ranging from constrained settings [19, 28, 32] to free-”
Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources (2015, neutral): “[27] built a query answering system based on a joint parse graph from text and videos.”
3D Human Activity Recognition with Reconfigurable Convolutional Neural Networks (2014, methodology): “Recently, the reconfigurable models were developed in the form of And-Or graphs [41, 22, 14, 28, 32], and yielded competitive performance in several challenging scenarios.”
Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning (2017, background): “Two recent works [ 5, 6 ] have proposed studying this task of visually-grounded visually-grounded dialog.”
IQA: Visual Question Answering in Interactive Environments (2017, background): “Other bodies of research involve the use of external knowledge bases [83, 84, 88] and explicit scene parsing [82] to answer visual questions.”
FVQA: Fact-Based Visual Question Answering (2016, neutral): “[13] built a query answering system based on a joint parse graph from text and videos.”
Measuring Machine Intelligence Through Visual Question Answering (2016, methodology): “To solve these two problems we propose the task of Visual Question Answering (VQA) [1, 18, 24, 29, 4, 17].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Initially, joint parsing of the videos and corresponding text to answer queries is explored (Tu et al., 2014).”
A Better Way to Attend: Attention With Trees for Video Question Answering (2018, background): “The “fill-in-the-blank” [20], [29], [40], [42] task requires a blank of a sentence to be filled with a word given the videos.”
Visual question answering: A survey of methods and datasets (2016, methodology): “Another early dataset was presented in [74] by Tu et al.”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “As an improvement over the model proposed by [1], [3] proposes a flexible encoder-decoder architecture using RNN, which covers a wide range of temporal information.”
Uncovering the Temporal Context for Video Question Answering (2017, background): “Video Question Answering and temporal structure reasoning Oneof the recentworks onvideo-basedquestion answering is Tu et al. (2014), which built a query answering system based on a joint parsing graph from both text and videos.”
Leveraging Video Descriptions to Learn Video Question Answering (2016, background): “ number of open-domain questions that can be used to train datademanding models such as deep learning models. Video-QA. In contrast to the Image-QA task, video-based QA is a much less explored task. (Tu et al. 2014) have studied joint parsing of videos and corresponding text to answer queries. (Tapa”