End-to-End Dense Video Captioning with Masked Transformer

By Luowei Zhou, Yingbo Zhou, Jason J. Corso, Richard Socher, Caiming Xiong. 2018

599 citations recorded by the source corpus.

Dense video captioning aims to generate text descriptions for all events in an untrimmed video. This involves both detecting and describing events. Therefore, all previous methods on dense video captioning tackle this problem by building two models, i.e. an event proposal and a captioning model, for these two sub-problems. The models are either trained separately or in alternation. This prevents direct influence of the language description to the event proposal, which is important for generating accurate descriptions. To address this problem, we propose an end-to-end transformer model for dense video captioning. The encoder encodes the video into appropriate representations. The proposal decoder decodes from the encoding with different anchors to form video event proposals. The captioning decoder employs a masking network to restrict its attention to the proposal event over the encoding feature. This masking network converts the event proposal to a differentiable mask, which ensures the consistency between the proposal and captioning during training. In addition, our model employs a self-attention mechanism, which enables the use of efficient non-recurrent structure during encoding and leads to performance improvements. We demonstrate the effectiveness of this end-to-end model on ActivityNet Captions and YouCookII datasets, where we achieved 10.12 and 6.58 METEOR score, respectively.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Compressive Transformers for Long-Range Sequence Modelling (2019, background): “, 2019), video captioning (Zhou et al., 2018), and a multitude of language understanding benchmarks (Devlin et al.”
VideoBERT: A Joint Model for Video and Language Representation Learning (2019, methodology): “, [15, 2, 10, 38, 39]) have trained models to analyse instructional videos, such as cooking.”
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, methodology): “On one hand, VideoBERT (Sun et al., 2019b), Unicoder-VL (Li et al., 2020a), VL-BERT (Su et al., 2020), UNITER (Chen et al., 2020), VLP (Zhou et al., 2018), ActBERT (Zhu and Yang, 2020) adopt a shared encoder approach, where the vision and text sequences are concatenated and input to a single…”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “Recognizing the objects “toddler”, “lady” and describing the independent action contents like “a toddler is crying” and “a lady picks the toddler up” in a video are now possible with advanced neural network models [14, 28, 63].”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “…since the release of the HowTo100M dataset [44], a large-scale instructional video dataset, there has been a flurry of works leveraging large-scale pretraining to improve video-text representations for tasks such as video question-answering [57], text-video retrieval [48] and video captioning [79].”
ActBERT: Learning Global-Local Video-Text Representations (2020, methodology): “We follow the setup from [54] that takes the video clips from YouCook2 [53] as input, and a transformer decoder is used to decode videos to captions.”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “YouCook2 focuses on cooking videos only, and the captions in MSR-VTT are very simple.”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “15 - EMT (Zhou et al., 2018b) V 0 - 4.”
Unified Vision-Language Pre-Training for Image Captioning and VQA (2019, methodology): “2019b) Video Single-stream Transformer+ Masked Transformer (Zhou et al. 2018) Zero-shot action classification Video captioning”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, methodology): “We follow the experimental setup from [40], where the ground truth video segmentations from YouCook2 are used to train a supervised model mapping video segments to captions.”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet (2021, background): “Recent works have explored to apply Transformers to various vision tasks: image classification [5, 12], object detection [3, 59, 56, 8, 34], segmentation [4, 40], image enhancement [4, 45], image generation [27], video processing [58, 51], and 3D point cloud processing [54].”
CvT: Introducing Convolutions to Vision Transformers (2021, background): “…a viable alternative to the convolutional neural networks (CNNs) in visual recognition tasks, such as classiﬁcation [11, 30], object detection [3, 45, 43, 8, 28], segmentation [33, 36], image enhancement [4, 40], image generation [24], video processing [42, 44] and 3D point cloud processing [12].”
Support-set bottlenecks for video-text representation learning (2020, background): “0 − − Masked Transformer (Zhou et al., 2018) 2 .”
COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis (2019, methodology): “, step localization, action segmentation, procedure segmentation [52], dense video caption [53] and visual grounding [25, 51].”
Grounded Video Description (2018, methodology): “For image captioning, we compare against two SotA methods, Neural Baby Talk (NBT) [15] and BUTD [1].”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, methodology): “In addition to image classification, Transformers have also been applied to other compute vision tasks, including object detection [3, 54, 50, 10], segmentation [41, 44], image enhancement [4, 46], image generation [28, 6], video processing [48, 53], and vision-language tasks [26, 35, 7, 33, 21, 20,”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, methodology): “…a single cross-modal encoder and concatenates the vision and text sequences as inputs, including VideoBERT (Sun et al., 2019b), Unicoder-VL (Li et al., 2020a), VL-BERT (Su et al., 2020), UNITER (Chen et al., 2020b), VLP (Zhou et al., 2018), ActBERT (Zhu and Yang, 2020) and VLM (Xu et al., 2021).”