End-to-end Generative Pretraining for Multimodal Video Captioning

By Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, Cordelia Schmid. 2022

201 citations recorded by the source corpus.

Recent video and language pretraining frameworks lack the ability to generate sentences. We present Multimodal Video Generative Pretraining (MV-GPT), a new pretraining framework for learning from unlabelled videos which can be effectively used for generative tasks such as multimodal video captioning. Unlike recent video-language pretraining frameworks, our framework trains both a multimodal video encoder and a sentence decoder jointly. To overcome the lack of captions in unlabelled videos, we leverage the future utterance as an additional text source and propose a bidirectional generation objective - we generate future utterances given the present mulitmodal context, and also the present utterance given future observations. With this objective, we train an encoder-decoder model end-to-end to generate a caption from raw pixels and transcribed speech directly. Our model achieves state-of the-art performance for multimodal video captioning on four standard benchmarks, as well as for other video understanding tasks such as VideoQA, video retrieval and action classification.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Cordelia Schmid
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Xiaogang Wang
  8. Andrew Zisserman

In Multimodal Learning

  1. Zhe Gan
  2. Lijuan Wang
  3. Zicheng Liu
  4. Jianfeng Wang
  5. Linjie Li
  6. Xiaowei Hu
  7. Zhengyuan Yang
  8. Jianfeng Gao

In Natural Language Generation

  1. Mohit Bansal
  2. Ming-Hsuan Yang
  3. Radu Soricut
  4. Arsha Nagrani
  5. Zhengyuan Yang
  6. Derek Hoiem
  7. Anurag Arnab
  8. Jie Lei

Citations in the corpus