Grounded Video Description

By Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J. Corso, Marcus Rohrbach. 2018

205 citations recorded by the source corpus.

Video description is one of the most challenging problems in vision and language understanding due to the large variability both on the video and language side. Models, hence, typically shortcut the difficulty in recognition and generate plausible sentences that are based on priors but are not necessarily grounded in the video. In this work, we explicitly link the sentence to the evidence in the video by annotating each noun phrase in a sentence with the corresponding bounding box in one of the frames of a video. Our dataset, ActivityNet-Entities, augments the challenging ActivityNet Captions dataset with 158k bounding box annotations, each grounding a noun phrase. This allows training video description models with this data, and importantly, evaluate how grounded or "true" such model are to the video they describe. To generate grounded captions, we propose a novel video description model which is able to exploit these bounding box annotations. We demonstrate the effectiveness of our model on our dataset, but also show how it can be applied to image description on the Flickr30k Entities dataset. We achieve state-of-the-art performance on video description, video paragraph description, and image description and demonstrate our generated sentences are better grounded in the video.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

ActBERT: Learning Global-Local Video-Text Representations (2020, background): “, video question answering [36, 10, 18, 54], video captioning [46, 52], textvideo retrieval [47, 41, 25], video grounding [50].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, background): “Another popular task is video captioning (Venugopalan et al., 2015; Pan et al., 2016; Gan et al., 2017; Zhou et al., 2018b, 2019), mostly benchmarking on Youtube2Text (Guadarrama et al., 2013), MSR-VTT (Xu et al., 2016), YouCook2 (Zhou et al., 2018a), ActivityNet Captions (Krishna et al., 2017) and…”
Unified Vision-Language Pre-Training for Image Captioning and VQA (2019, methodology): “Existing works (Zhou et al. 2019; Lu et al. 2018) regard region object labels (probabilities) ( C i ) as an important auxiliary to enrich image region features and here we follow a similar design.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “ActivityNet [27] is the most popular video challenge, evaluated for localisation, dense captioning [31] and object detection [32].”
Transformers in Vision: A Survey (2021, background): “, language and vision [17], [199]), an interesting question to consider is whether the pre-training process leveraging rich labels on a small dataset speedup its learning.”
Connecting Vision and Language with Localized Narratives (2019, methodology): “In the video domain, ActivityNet-Entities [70] adds visual grounding to the ActivityNet Captions, also in two stages where boxes were drawn a posteriori.”
TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, methodology): “Recent works on grounded image captioning and video captioning [46, 22, 47] show better performance can be achieved by explicitly supervising the attention.”
XGPT: Cross-modal Generative Pre-Training for Image Captioning (2020, neutral): “9 GVD [33] 62.”
Crossing the Format Boundary of Text and Boxes: Towards Unified Vision-Language Modeling (2021, methodology): “For grounded captioning [38, 70, 72], we benchmark on the Flickr30k Entities dataset [42].”
More Grounded Image Captioning by Distilling Image-Text Matching Model (2020, methodology): “Most of them supervise the learning process by the attention module [28, 65, 36].”
Grounding Object Detections With Transcriptions (2019, methodology): “, 2015) or activtiyNet (Zhou et al., 2019), as opposed to thie this work where we use How2 an “in the wild” collected dataset.”
ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes (2020, background): “Recent work grounds noun phrases in image cap-tions, such as in Flickr30KEntities [49] and ActivityNetEntities [73] in videos.”
Spatio-Temporal Graph for Video Captioning With Knowledge Distillation (2020, background): “Similar to previous work [8, 11, 34, 43, 49, 51], we also directly model the frame sequence through a separate scene branch.”
Multi-interaction Network with Object Relation for Video Question Answering (2019, unknown): “[43] link the sentence to the video evidence by annotating each noun phrase with the corresponding bounding box in video frames.”
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning (2021, background): “While many works (Zhou et al., 2019; 2018; Gan et al., 2015) have been studying on the problems of understanding human actions and activities (e.g., running, cooking, cleaning) in videos, our work’s primary goal is to design a uniﬁed framework for learning physical object and event concepts (e.g.,…”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “Further, extending earlier research, some approaches (Zhou et al., 2019) have explicitly linked the sentence to a corresponding bounding box in one of the frames of a video by annotating each of the noun phrases observed in the sentence.”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, methodology): “Our model also excels at generating paragraphs of text describing the video: without using ground-truth event proposals at inference time, our model outperforms all prior approaches including those that rely on such proposals [50,76,128].”
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling (2021, methodology): “The word and box alignments are then separately generated as additional predictions, such as the relevance score [23,78,80,45].”