MovieNet: A Holistic Dataset for Movie Understanding

By Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, Dahua Lin. European Conference on Computer Vision, 2020

319 citations recorded by the source corpus.

Recent years have seen remarkable advances in visual understanding. However, how to understand a story-based long video with artistic styles, e.g. movie, remains challenging. In this paper, we introduce MovieNet -- a holistic dataset for movie understanding. MovieNet contains 1,100 movies with a large amount of multi-modal data, e.g. trailers, photos, plot descriptions, etc. Besides, different aspects of manual annotations are provided in MovieNet, including 1.1M characters with bounding boxes and identities, 42K scene boundaries, 2.5K aligned description sentences, 65K tags of place and action, and 92K tags of cinematic style. To the best of our knowledge, MovieNet is the largest dataset with richest annotations for comprehensive movie understanding. Based on MovieNet, we set up several benchmarks for movie understanding from different angles. Extensive experiments are executed on these benchmarks to show the immeasurable value of MovieNet and the gap of current approaches towards comprehensive movie understanding. We believe that such a holistic dataset would promote the researches on story-based long video understanding and beyond. MovieNet will be published in compliance with regulations at this https URL.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0732df185bdfcb9c908ec30bb441252593f58875

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…Image2Latex, Synthetic Handwritten OCR, Synthetic Infographic2Markdown KVQA [207], A-OKVQA [205], ViQuAE [123], iNaturalist2018 [237], MovieNet [95], ART500K [176], KonIQ-10K [91], IconQA [167], VisualMRC [225], ChemVLM Data [129], ScienceQA [165], AI2D [109], Knowledge TQA [110], Wikipedia-QA…”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “Further, MAD [43] and several other movie-based datasets [26, 47, 53] do not release any video data because of copyright issues.”
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (2023, methodology): “In this study, we construct the training set mainly from [5, 21, 32, 39], as illustrated in Section 3.3.”
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (2023, background): “Similar with the above coarse-grained recognition, the images of these subtasks are from publicly available datasets [25, 37, 38, 55, 46] and all of the instructions are manually designed.”
MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, unknown): “Similarly, LLaMA-Vid [22] created a movie question-answering dataset based on MovieNet [13].”
Condensed Movies: Story Based Retrieval with Contextual Embeddings (2020, background): “9 Descriptions, graphs MovieQA (video)[14] 140 381 VQA MovieScenes [33] 150 250 Scene segmentations LSMDC [2] 202 158 Captions MSA [3] 327 516 Plots MovieNet[5] 1,100 2,000 Plots, action tags, character IDs”
Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets (2020, background): “In contrast, the distribution of different object categories typically exhibit a long tail in practical contexts while individual images can generally be associated with multiple semantic labels [24,23,15,35].”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “, MovieQA [79], MVAD [81], MPII-MD [66], and MovieNet [32]) from different sources such as YouTube [14,58,105], Twitter [5–8], Flirck [5, 6], and internet [9].”