Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning

By Zhiyuan Fang, Tejas Gokhale, Pratyay Banerjee, Chitta Baral, Yezhou Yang. 2020

68 citations recorded by the source corpus.

Captioning is a crucial and challenging task for video understanding. In videos that involve active agents such as humans, the agent's actions can bring about myriad changes in the scene. These changes can be observable, such as movements, manipulations, and transformations of the objects in the scene -- these are reflected in conventional video captioning. However, unlike images, actions in videos are also inherently linked to social and commonsense aspects such as intentions (why the action is taking place), attributes (such as who is doing the action, on whom, where, using what etc.) and effects (how the world changes due to the action, the effect of the action on other agents). Thus for video understanding, such as when captioning videos or when answering question about videos, one must have an understanding of these commonsense aspects. We present the first work on generating \textit{commonsense} captions directly from videos, in order to describe latent aspects such as intentions, attributes, and effects. We present a new dataset "Video-to-Commonsense (V2C)" that contains 9k videos of human agents performing various actions, annotated with 3 types of commonsense descriptions. Additionally we explore the use of open-ended video-based commonsense question answering (V2C-QA) as a way to enrich our captions. We finetune our commonsense generation models on the V2C-QA task where we ask questions about the latent aspects in the video. Both the generation task and the QA task can be used to enrich video captions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Natural Language Generation

Citations in the corpus

NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “Other works like Motivation [46], VCR [58] and V2C [10] may also take causality into consideration, either for visual description or QA.”
Compressing Visual-linguistic Model via Knowledge Distillation (2021, background): “, image/video captioning [3, 79, 70, 76, 26, 15], VQA [21, 4, 19], textual grounding [56, 16, 24, 82], etc.”
Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, methodology): “[36] proposed an open-ended VQA framework for videos using commonsense reasoning in the language part, where questions are asked about effects, intents, and attributes.”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “[13] Zhiyuan Fang, Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang.”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, unknown): “Each paper was published between 2018 and 2020 at ACL, NAACL, or EMNLP, and we exclude papers that use AMT to evaluate more wellestablished generation tasks like machine translation, or summarization.3 Unlike previous surveys of evaluating generated text (Çelikyilmaz et al., 2020; van der Lee et al.”
The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning (2022, background): “Some prior work has explored commonsense reasoning tasks posed over videos, which usually have more input available than a single frame [12,13,19,20,30,31,33,63,74,78] (inter alia).”
Injecting Semantic Concepts into End-to-End Image Captioning (2021, methodology): “Most previous captioning models unani-mously [4,15,21,25,58,63,66,73,77] use detector based visual encoder like Faster-RCNN [57] to extract visual features, and apply decoders like RNN, LSTM or Transformer for caption generation.”