Annotating Objects and Relations in User-Generated Videos

By Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, Tat-seng Chua. 2019

213 citations recorded by the source corpus.

Understanding the objects and relations between them is indispensable to fine-grained video content analysis, which is widely studied in recent research works in multimedia and computer vision. However, existing works are limited to evaluating with either small datasets or indirect metrics, such as the performance over images. The underlying reason is that the construction of a large-scale video dataset with dense annotation is tricky and costly. In this paper, we address several main issues in annotating objects and relations in user-generated videos, and propose an annotation pipeline that can be executed at a modest cost. As a result, we present a new dataset, named VidOR, consisting of 10k videos (84 hours) together with dense annotations that localize 80 categories of objects and 50 categories of predicates in each video. We have made the training and validation set public and extendable for more tasks to facilitate future research on video object and relation recognition.

Subfields: Computer Vision and Pattern Recognition, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Extraction

Citations in the corpus

NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “Additionally, it could be interesting to incorporate explicit relation information, as NExT-QA’s videos are sourced from the VidOR [38] dataset where relation annotations already exist and provide a rich source of information to be leveraged.”
MMGCN: Multi-modal Graph Convolution Network for Personalized Recommendation of Micro-video (2019, methodology): “First, we would construct multi-modal knowledge graph to present objects and relations between them in microvideos [31], and then use it into MMGCN to model finer-grained content analysis.”
Video Instruction Tuning With Synthetic Data (2024, unknown): “…11 video benchmarks. conducted tests across various video captioning , video open-ended question-answering and video multiple-choice question-answering benchmarks, including ActivityNet-QA (Yu et al., 2019), which features human-annotated action-related QA pairs from the ActivityNet dataset.”
LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, unknown): “Note that the training set for LLaVA-Video includes six datasets: LLaVA-Video-178K, LLaVA-Hound (Zhang et al., 2024d), NExT-QA (Xiao et al., 2021), ActivityNet-QA (Yu et al., 2019), PerceptionTest (Pătrăucean et al., 2023), and image data from LLaVA-OneVision (Li et al., 2024c).”
OtterHD: A High-Resolution Multi-modality Model (2023, methodology): “Specifically, the collection comprises 172 QA pairs from 108 images in EpicKitchen [14], 80 QAs from 38 images in Ego4D [22], and 31 QA pairs from 20 images in VidOR [46].”
Visual Relation Grounding in Videos (2020, background): “Considering that annotating fine-grained regionrelation pairs in videos is complicated and labor-intensive [30], we define the task as weakly-supervised grounding where only video-relation correspondences are available.”
PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “NExT-QA [59]: We focus on the Open-Ended QA task, which consists of 52,044 question-answer pairs for a total of 5,440 videos (sampled from the VidOr dataset[98]).”
Beyond Short-Term Snippet: Video Relation Detection With Spatio-Temporal Global Context (2020, methodology): “[33] also proposed a baseline for VidVRD task, decomposing one video into segments and merging relationship predictions in neighbouring segments through a greedy association algorithm.”
Video Graph Transformer for Video Question Answering (2022, background): “Recently, graphs constructed over object-level representations [19,36,47,60] have demonstrated superior performance, especially on benchmarks that emphasize visual relation reasoning [20,49,50,59].”