Dense-Captioning Events in Videos

By Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, Juan Carlos Niebles. 2017

1,593 citations recorded by the source corpus.

Most natural videos contain numerous events. For example, in a video of a “man playing a piano”, the video might also contain “another man dancing” or “a crowd clapping”. We introduce the task of dense-captioning events, which involves both detecting and describing events in a video. We propose a new model that is able to identify all events in a single pass of the video while simultaneously describing the detected events with natural language. Our model introduces a variant of an existing proposal module that is designed to capture both short as well as long events that span minutes. To capture the dependencies between the events in a video, our model introduces a new captioning module that uses contextual information from past and future events to jointly describe all events. We also introduce ActivityNet Captions, a large-scale benchmark for dense-captioning events. ActivityNet Captions contains 20k videos amounting to 849 video hours with 100k total descriptions, each with its unique start and end time. Finally, we report performances of our model for dense-captioning events, video retrieval and localization.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “secutive video captions from ActivityNet Captions (Krishna et al., 2017; Heilbron et al., 2015) and the Large Scale Movie Description Challenge (LSMDC; Rohrbach et al.”
HellaSwag: Can a Machine Really Finish Your Sentence? (2019, methodology): “We start by including video captions from the ActivityNet Captions dataset (Krishna et al., 2017).”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “The examples are mined from Wikihow and Activitynet Captions [29] dataset.”
VideoBERT: A Joint Model for Video and Language Representation Learning (2019, methodology): “, [16, 2, 10, 38, 39]) have trained models to analyse instructional videos, such as cooking.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “These applications include automatic image and video captioning [20, 36, 60, 61], visual question answering [9, 29, 51, 63], visual content retrieval based on textual queries [32, 56, 63], temporal localization of events in videos using natural language [15, 26] or video summarization with natural l”
All in One: Exploring Unified Video-Language Pre-Training (2022, methodology): “Datasets: MSRVTT [42], DiDeMo [3], ActivityNet Captions [18] are utilized for this task.”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “Popular VidL tasks were introduced, such as text-to-video retrieval [1–5], video question answering [6–9], text-based video moment retrieval [2, 4, 10, 11], and video captioning [1, 3, 12, 13].”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “[96] use a pretrained dense image captioner [49] to provide auxiliary labels for web how-to videos.”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “, “Why is the toddler crying?”, “How did the lady react after the toddler fell?”), which lies at the core of human intelligence [39], remains a great challenge for computational models and is also much less explored by existing video understanding tasks [22, 37, 49, 52, 56].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, background): “…captioning (Venugopalan et al., 2015; Pan et al., 2016; Gan et al., 2017; Zhou et al., 2018b, 2019), mostly benchmarking on Youtube2Text (Guadarrama et al., 2013), MSR-VTT (Xu et al., 2016), YouCook2 (Zhou et al., 2018a), ActivityNet Captions (Krishna et al., 2017) and VATEX (Wang et al., 2019).”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “Different from existing works (Sun et al., 2019b,a; Krishna et al., 2017; Zhou et al., 2018a,b; Shi et al., 2019; Palaskar et al., 2019; Hessel et al., 2019) which only use video signal, recent works (Shi et al., 2019; Palaskar et al., 2019; Hessel et al., 2019) study the multimodal captioning by…”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, background): “, [30]), though they may appear as different variants due to the difference of the corpora, e.”
Towards Automatic Learning of Procedures From Web Instructional Videos (2017, methodology): “The output procedure segments of ProcNets can be applied for other tasks, including full agent-based procedure learning (Yu and Siskind 2013) or smaller-scale video description generation (Yu et al. 2016; Krishna et al. 2017).”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “While current literature on language+vision focuses on captioning and question answering for isolated instances of Internet data [12, 35, 119, 228], NLQ is motivated by queries about the camera wearer’s own visual experience and operates over long-term observations.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “ActivityNet [27] is the most popular video challenge, evaluated for localisation, dense captioning [31] and object detection [32].”
TVQA: Localized, Compositional Video Question Answering (2018, methodology): “…features (upper) and subtitle (bottom) streams. brown door, gold sign, red sign, woman, white shorts, green sweater, man, blue shirt, white basket, woman, gray pants, gray door, standing man, gray shirt, black pants Genome (Krishna et al., 2017) to detect object and attribute regions in each frame.”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, methodology): “…including image cap-tioning datasets, such as Flickr30k (Young et al., 2014), MSCOCO (Lin et al., 2014), Conceptual Captions (Sharma et al., 2018), as well as video captioning datasets including LSMDC (Rohrbach et al., 2017), ActivityNet (Krishna et al., 2017), and VATEX (Wang et al., 2019b).”
End-to-End Dense Video Captioning with Masked Transformer (2018, neutral): “The other way is to alternate training [19] between the two modules, i.”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, methodology): “Popular video-andlanguage tasks include text-to-video retrieval [75, 28, 54], video captioning [75, 82, 28, 54, 40], video question answering [74, 23, 33], and moment retrieval [2, 18, 35].”