TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval

By Jie Lei, Licheng Yu, Tamara L. Berg, Mohit Bansal. 2020

361 citations recorded by the source corpus.

We introduce TV show Retrieval (TVR), a new multimodal retrieval dataset. TVR requires systems to understand both videos and their associated subtitle (dialogue) texts, making it more realistic. The dataset contains 109K queries collected on 21.8K videos from 6 TV shows of diverse genres, where each query is associated with a tight temporal window. The queries are also labeled with query types that indicate whether each of them is more related to video or subtitle or both, allowing for in-depth analysis of the dataset and the methods that built on top of it. Strict qualification and post-annotation verification tests are applied to ensure the quality of the collected data. Further, we present several baselines and a novel Cross-modal Moment Localization (XML ) network for multimodal moment retrieval tasks. The proposed XML model uses a late fusion design with a novel Convolutional Start-End detector (ConvSE), surpassing baselines by a large margin and with better efficiency, providing a strong starting point for future work. We have also collected additional descriptions for each annotated moment in TVR to form a new multimodal captioning dataset with 262K captions, named TV show Caption (TVC). Both datasets are publicly available. TVR: this https URL, TVC: this https URL.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Multimodal Learning

Citations in the corpus

Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “Based on this, we use a query encoder (Lei et al., 2020), consisting of a self-attention layer, two linear layers and a LN layer, to obtain the final query vector q ∈ Rd from Wcrosssq .”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “VATEX (Wang et al., 2019b), and TVC (Lei et al., 2020) (results in supplementary materials.).”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, background): “A wide range of tasks based on real-life videos have been designed to test such ability, including text-to-video retrieval [75, 28, 54], video captioning [54, 75, 82], video question answering [74, 23, 33, 34], and video moment retrieval [2, 18, 35].”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, methodology): “For VCMR ( i.e. , TVR and How2R), we additionally require that the predicted span of a correct prediction has a high overlap with the ground-truth moment.”
What Is More Likely to Happen Next? Video-and-Language Future Event Prediction (2020, methodology): “Various datasets and tasks have been introduced in this area, such as video captioning (Xu et al., 2016; Rohrbach et al., 2017; Wang et al., 2019; Lei et al., 2020c), video QA (Tapaswi et al., 2016; Jang et al., 2017; Lei et al., 2018), and moment retrieval (Hendricks et al., 2017; Gao et al.,…”
QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries (2021, background): “Moment retrieval [13, 7, 18] requires localizing moments from a video given a natural language query.”
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization (2021, methodology): “For YouCook2, we follow previous work (Lei et al., 2020a; Ging et al., 2020) to evaluate performance at paragraph-level, where single segment captions are concatenated as a paragraph for evaluation.”
Violin: A Large-Scale Dataset for Video-and-Language Inference (2020, neutral): “Video-and-Language Research With the emergence of large-scale video datasets [6, 1, 29, 11, 58], several videoand-language tasks have been proposed, such as video captioning [21, 56, 62, 18, 33, 16, 47, 59], localizing video segments from natural language queries [19, 3, 8, 37], video reasoning [65]”
VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs (2021, methodology): “TVC [28] is a recently introduced benchmark for video captioning.”
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning (2021, background): “Although it has shown great success in video-and-language understanding tasks, such as video question answering [24] and text-tovideo retrieval [25, 56], it remains unclear whether these sparsely sampled video frames are sufficient to generate rich and descriptive captions.”
MIMIC-IT: Multi-Modal In-Context Instruction Tuning (2023, methodology): “By organizing drama clips from [24] to analyze character relationships and motivations, we aim to challenge VLMs to move beyond mere perception and demonstrate their reasoning capabilities within the context of TV show narratives.”
Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023, neutral): “Otter: The sport involves players running and trying to catch the ball while others are standing in the grass, which can lead to collisions or accidents.”
DoRA: Weight-Decomposed Low-Rank Adaptation (2024, background): “This is because pre-trained weights already contain substantial knowledge suitable for downstream tasks, and a larger magnitude or direction alteration alone is sufficient (Li et al., 2020) for video question answering, TVC (Lei et al., 2020) and YC2C (Zhou et al., 2018) for video captioning.”
Video Understanding as Machine Translation (2020, methodology): “We measure the captioning performance of our system on YouCook2 [64], TVC [25], and MSRVTT [61].”
CLIP4Caption ++: Multi-CLIP for Video Caption (2021, methodology): “We report the single model result and ensemble result on the public test set and validation test set of TVC [4], YC2C [14], VATEX [13] datasets in Table1.”