End-to-end Generative Pretraining for Multimodal Video Captioning
By Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, Cordelia Schmid. 2022
201 citations recorded by the source corpus.
Recent video and language pretraining frameworks lack the ability to generate sentences. We present Multimodal Video Generative Pretraining (MV-GPT), a new pretraining framework for learning from unlabelled videos which can be effectively used for generative tasks such as multimodal video captioning. Unlike recent video-language pretraining frameworks, our framework trains both a multimodal video encoder and a sentence decoder jointly. To overcome the lack of captions in unlabelled videos, we leverage the future utterance as an additional text source and propose a bidirectional generation objective - we generate future utterances given the present mulitmodal context, and also the present utterance given future observations. With this objective, we train an encoder-decoder model end-to-end to generate a caption from raw pixels and transcribed speech directly. Our model achieves state-of the-art performance for multimodal video captioning on four standard benchmarks, as well as for other video understanding tasks such as VideoQA, video retrieval and action classification.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Cordelia Schmid
- Ming-Hsuan Yang
- Jitendra Malik
- Xiaogang Wang
- Andrew Zisserman
In Multimodal Learning
In Natural Language Generation
- Mohit Bansal
- Ming-Hsuan Yang
- Radu Soricut
- Arsha Nagrani
- Zhengyuan Yang
- Derek Hoiem
- Anurag Arnab
- Jie Lei
Citations in the corpus
- GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “8 [127] [122] [30] [120] [5] [3] [5] [69] [92] [100] [100] [103] [125] [78]”
- Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners (2022, background): “Meanwhile, unified video-language models [34, 48] that are capable of language decoding still rely heavily on finetuning using a large number of manually annotated video-text pairs, therefore cannot adapt quickly to unseen tasks.”
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, background): “…data (images or videos) and linguistic signals (words or sentences) and can be applied to several downstream applications, including text-driven image or video generation [4, 8, 29, 57, 59, 61, 65, 83, 92], captioning [2, 36, 37, 43, 63, 81], VQA [14, 38, 53, 88, 94] and re-1 trieval [17, 39, 49].”
- Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “[89] Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid.”
- PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “5§ GIT2 [9] Flamingo [10] PDVC [62] VINDLU [63] GIT2 [9] MV-GPT [64] Flamingo 32shot [10]”
- GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions (2023, methodology): “However, due to its success in text processing, users expect its integration with other modalities, such as speech recognition, video summarization, and image or video captioning [199].”
- Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “83 MV-GPT [192] arXiv-2022 image-text BERT MLM, LG Pre-train both a multi-modal video encoder and a sentence decoder jointly.”
- HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training (2022, methodology): “On MSRVTT Caption, our method surpasses SoTA method MV-GPT [40] by 2.5% CIDEr.”