Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
By Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, Cordelia Schmid. Computer Vision and Pattern Recognition, 2023
401 citations recorded by the source corpus.
In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale. The Vid2Seq architecture augments a language model with special time tokens, allowing it to seamlessly predict event boundaries and textual descriptions in the same output sequence. Such a unified model requires large-scale training data, which is not available in current annotated datasets. We show that it is possible to leverage unlabeled narrated videos for dense video captioning, by reformulating sentence boundaries of transcribed speech as pseudo event boundaries, and using the transcribed speech sentences as pseudo event captions. The resulting Vid2Seq model pretrained on the YT-Temporal-1B dataset improves the state of the art on a variety of dense video captioning benchmarks including YouCook2, ViTT and ActivityNet Captions. Vid2Seq also generalizes well to the tasks of video paragraph captioning and video clip captioning, and to few-shot settings. Our code is publicly available at [1].
Subfields: Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/0938d0ccc1c633fa0f8c067d914358b1ef53a44b
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Ming-Hsuan Yang
- Sergey Tulyakov
- Hsin-Ying Lee
- Jian Ren
- Aliaksandr Siarohin
- Luc Van Gool
- Jenq-Neng Hwang
- Xu Sun
Citations in the corpus
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (2023, background): “There have been miscellaneous time-sensitive video tasks, including temporal video grounding [27, 49], dense video captioning [46, 52, 62], video summarization [1, 8, 41, 58], video highlight detection [15, 30], and step localization [5, 39, 55], etc.”
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, background): “Nevertheless, it would limit the content diversity within a single video and also reduce average video duration, which might be hurtful to the downstream tasks, such as long video generation [9] and dense video captioning [71, 81].”
- Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “…demonstrated by models such as Video-Chat [47], Video-ChatGPT [107], Video-LLaMA [48], LLaMA-VID [108], MotionGPT [109], LAVENDER [110], MovieChat [111], Vid2Seq [112], VideoLLM [113], and VTimeLLM [114], shows significant promise for the development of models tailored to medical applications.”
- Retrieval-Augmented Generation for Large Language Models: A Survey (2023, methodology): “Vid2Seq augments language models with specialized temporal markers, facilitating the prediction of event boundaries and textual descriptions within a unified output sequence [ Yang et al. , 2023a ] .”
- VTimeLLM: Empower LLM to Grasp Video Moments (2023, background): “9 } , and calculate captioning metrics based on these matched pairs [28].”
- MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “, video question answering [37, 38], video captioning [34, 98], and human action recognition [36, 62, 63]).”