UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation

By Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Xilin Chen, Ming Zhou. 2020

412 citations recorded by the source corpus.

With the recent success of the pre-training technique for NLP and image-linguistic tasks, some video-linguistic pre-training works are gradually developed to improve video-text related downstream tasks. However, most of the existing multimodal models are pre-trained for understanding tasks, leading to a pretrain-finetune discrepancy for generation tasks. This paper proposes UniVL: a Unified Video and Language pre-training model for both multimodal understanding and generation. It comprises four components, including two single-modal encoders, a cross encoder, and a decoder with the Transformer backbone. Five objectives, including video-text joint, conditioned masked language model (CMLM), conditioned masked frame model (CMFM), video-text alignment, and language reconstruction, are designed to train each of the components. We further develop two pre-training strategies, stage by stage pre-training (StagedP) and enhanced video representation (EnhancedV), to make the training process of the UniVL more effective. The pre-train is carried out on a sizeable instructional video dataset HowTo100M. Experimental results demonstrate that the UniVL can learn strong video-text representation and achieves state-of-the-art results on five downstream tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “…pre-training has brought leaping advances in vision+language understanding tasks such as VQA and VCR, with great potential in extending to visual captioning [84, 70], visual dialog [47, 68], vision-language navigation [18, 43], as well as video-and-language representation learning [62, 61, 41, 31].”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, neutral): “Similar to our work are VLMs able to generate text in an autoregressive manner [19, 25, 45, 67, 116].”
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, methodology): “CBT (Sun et al., 2019a), HERO (Li et al., 2020b), VideoAsMT (Korbar et al., 2020) and UniVL (Luo et al., 2020) adopt multi-task learning (MTL) to learn retrieval tasks on video and text encoders.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, positive): “Methods using such readily-available data have shown signiﬁcant improvements on several tasks including video retrieval, action localization, action recognition and video captioning [26, 58, 59, 60, 74, 75, 103], sometimes outperforming fully-supervised baselines.”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, background): “, 2019a) and UniViLM (Luo et al., 2020) are the only existing works exploring this space.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[77] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, methodology): “…result on Youcook2 dataset R@1 ↑ R@5 ↑ R@10 ↑ S UPERVISED HGLMM(Klein et al., 2015) 4.6 14.3 21.6 Coot(Ging et al., 2020) 16.7 40.2 52.3 UniVL (FT-Joint)(Luo et al., 2020) 22.2 52.2 66.2 VideoCLIP (Fine-tuned) 32.2 62.6 75.0 Z ERO - SHOT Random 0.0 0.2 0.3 HowTo100M(Miech et al., 2019) 6.1 17.3…”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, methodology): “Recently, the pre-trained models (Zhu and Yang, 2020; Amrani et al., 2021; Luo et al., 2020; Li et al., 2020; Miech et al., 2020; Gabeur et al., 2020; Patrick et al., 2021; Lei et al., 2021; Dzabraev et al., 2021; Liu et al., 2021) have dominated the leaderboard of the video-text retrieval with…”
VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs (2021, background): “The benefit of this solution is that it opens up the possibility to carry out multimodal fusion by means of powerful language encoders such as T5 [40] without the need to design specialized cross-modal network modules [33, 29, 57, 35] or to resort to pretext tasks to learn to combine the different i”
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval (2022, background): “[30,28,50] adopts MFM, which masks video frames and recovers the masked frames to the features encoded from an off-the-shelf video feature extraction network [49].”
Disentangled Representation Learning for Text-Video Retrieval (2022, background): “With the great success of NLP pre-training [8,27,22,36,37], Vision-Language Pre-Training (VLPT) has received increasing attention [23,21,24,28,2].”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, methodology): “To quantitatively evaluate the quality of output captions, we follow the common protocols [41, 48, 78] and report BLEU-4 [54], ROGUE-L [40], METEOR [5], and CIDEr [69].”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “Similarly, multiple works utilize pretrained LMs for generating text descriptions of images or videos in image captioning (VisualGPT [120]) and video captioning (VideoBERT [121], Unified VLP [122], UniVL [123]).”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, background): “Although video BERT approaches have been proposed to learn the correspondence between texts and video frames [27, 35, 49, 57], we are the first to integrate recurrence into BERT to learn partiallyobservable and temporal-dependent inputs.”