Violin: A Large-Scale Dataset for Video-and-Language Inference

By Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, Jingjing Liu. 2020

79 citations recorded by the source corpus.

We introduce a new task, Video-and-Language Inference, for joint multimodal understanding of video and text. Given a video clip with aligned subtitles as premise, paired with a natural language hypothesis based on the video content, a model needs to infer whether the hypothesis is entailed or contradicted by the given video clip. A new large-scale dataset, named Violin (VIdeO-and-Language INference), is introduced for this task, which consists of 95,322 video-hypothesis pairs from 15,887 video clips, spanning over 582 hours of video. These video clips contain rich content with diverse temporal dynamics, event shifts, and people interactions, collected from two sources: (i) popular TV shows, and (ii) movie clips from YouTube channels. In order to address our new multimodal inference task, a model is required to possess sophisticated reasoning skills, from surface-level grounding (e.g., identifying objects and characters in the video) to in-depth commonsense reasoning (e.g., inferring causal relations of events in the video). We present a detailed analysis of the dataset and an extensive evaluation over many strong baselines, providing valuable insights on the challenges of this new task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Zhe Gan
3. Yejin Choi
4. Jianfeng Gao
5. Linjie Li
6. Jingjing Liu
7. Lijuan Wang
8. Zicheng Liu

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Mohit Bansal
3. Luc Van Gool
4. Dacheng Tao
5. Cordelia Schmid
6. Trevor Darrell
7. Yi Yang
8. Jianfeng Gao

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Linjie Li
4. Mohit Bansal
5. Zicheng Liu
6. Jingjing Liu
7. Yu Cheng
8. Jianfeng Gao

In Evaluation & Benchmarks

1. Mohit Bansal
2. Yejin Choi
3. Chandra Bhagavatula
4. Rowan Zellers
5. Jie Lei
6. Jena D. Hwang
7. X. Wang
8. Zhe Gan

Citations in the corpus

• Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “, 2019) and a large-scale TV dataset (containing 660k TV episodes spanning different genres) (Lei et al., 2018, 2019, 2020; Liu et al., 2020).”
• VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, methodology): “To construct a comprehensive evaluation benchmark, we include recent datasets collected on multi-channel videos: TVR [35], How2R [36], TVQA [32], How2QA [36], VIOLIN [40], VLEP [34] and TVC [35].”
• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “[243] in year 2020, which aims at understanding the video and text multimodal data.”
• The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning (2022, background): “Some prior work has explored commonsense reasoning tasks posed over videos, which usually have more input available than a single frame [12,13,19,20,30,31,33,63,74,78] (inter alia).”