What Is More Likely to Happen Next? Video-and-Language Future Event Prediction

By Jie Lei, Licheng Yu, Tamara L. Berg, Mohit Bansal. 2020

85 citations recorded by the source corpus.

Given a video with aligned dialogue, people can often infer what is more likely to happen next. Making such predictions requires not only a deep understanding of the rich dynamics underlying the video and dialogue, but also a significant amount of commonsense knowledge. In this work, we explore whether AI models are able to learn to make such multimodal commonsense next-event predictions. To support research in this direction, we collect a new dataset, named Video-and-Language Event Prediction (VLEP), with 28,726 future event prediction examples (along with their rationales) from 10,234 diverse TV Show and YouTube Lifestyle Vlog video clips. In order to promote the collection of non-trivial challenging examples, we employ an adversarial human-and-model-in-the-loop data collection procedure. We also present a strong baseline incorporating information from video, dialogue, and commonsense knowledge. Experiments show that each type of information is useful for this challenging task, and that compared to the high human performance on VLEP, our model provides a good starting point but leaves large room for future work. Our dataset and code are available at: this https URL

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “The completion results (example below on the right) are generative, and are more broad than binary event classification [82].”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “We report results on 12 video reasoning tasks: TVQA [64], TVQA(+) [65], VLEP [66], MSRVTT-QA [117], MSRVTT-Multichoice [121], LSMDC-Multichoice, LSMDC ﬁll-in-the-blank QA [110, 92], ActivityNetQA [122, 45], TGIFQA [52], and DramaQA [23].”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, methodology): “To construct a comprehensive evaluation benchmark, we include recent datasets collected on multi-channel videos: TVR [35], How2R [36], TVQA [32], How2QA [36], VIOLIN [40], VLEP [34] and TVC [35].”
On the Diversity and Limits of Human Explanations (2021, background): “Lei et al. (2020) future event prediction we also require them to provide a rationale as to why it is more or less likely Da et al. (2020) harm of manip- ulated images For each question we require annotators to provide both an answer to the question and a rationale (e.g. the physical change in the…”
The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning (2022, background): “Some prior work has explored commonsense reasoning tasks posed over videos, which usually have more input available than a single frame [12,13,19,20,30,31,33,63,74,78] (inter alia).”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “[72] Jie Lei, Licheng Yu, Tamara Berg, and Mohit Bansal.”
QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries (2021, methodology): “Following [6, 19], we start with user-created lifestyle vlog videos on YouTube.”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “Other examples of datasets in both classes include predicting future events in videos ( VLEP ; Lei et al., 2020) and answering commonsense questions about images ( VCR ; Zellers et al., 2019a).”
Revisiting the “Video” in Video-Language Understanding (2022, background): “Standard video-language tasks include both discriminative tasks, such as video question answering [12, 19, 27, 28, 30, 54, 56, 58, 59], text-to-video/moment retrieval [16,23,43,56,62], and generative tasks, such as video cap-tioning [4, 23] and open-ended VQA [54, 56].”