VideoBERT: A Joint Model for Video and Language Representation Learning

By Chen Sun, Austin Myers, Carl Vondrick, Kevin P. Murphy, Cordelia Schmid. 2019

1,420 citations recorded by the source corpus.

Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube. Whereas most existing approaches learn low-level representations, we propose a joint visual-linguistic model to learn high-level features without any explicit supervision. In particular, inspired by its recent success in language modeling, we build upon the BERT model to learn bidirectional joint distributions over sequences of visual and linguistic tokens, derived from vector quantization of video data and off-the-shelf speech recognition outputs, respectively. We use VideoBERT in numerous tasks, including action classification and video captioning. We show that it can be applied directly to open-vocabulary classification, and confirm that large amounts of training data and cross-modal information are critical to performance. Furthermore, we outperform the state-of-the-art on video captioning, and quantitative results verify that the model learns high-level semantic features.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “…the output of a CNN using self-attention, e.g. for object detection (Hu et al., 2018; Carion et al., 2020), video processing (Wang et al., 2018; Sun et al., 2019), image classiﬁcation (Wu et al., 2020), unsupervised object discovery (Locatello et al., 2020), or uniﬁed text-vision tasks (Chen…”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “Multi-modal learning methods that combine vision and audio [42, 9, 46, 3], and vision and text [57] achieve better performance than unimodal baselines.”
VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “VideoBERT (Sun et al., 2019) transforms a video into spoken words paired with a series of images and applies a Transformer to learn joint representations.”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, methodology): “Most related to our approach is concurrent work on learning joint representations between video and language [29].”
Learning Visual Representations with Caption Annotations (2020, background): “Building on the recent advances in natural language processing [14,59], several works have fine-tuned BERT [14] to fuse visual and textual information [40,55,56,57,70] for VL tasks.”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Motivated by pre-trained models like BERT [1] and ERNIE [2] which have significantly improved the performance on many NLP tasks, researchers ( [3] [4] [5] [6] [7] [8] [9] [10]) have noticed the importance of pre-training for vision-language tasks, e.”
Self-Supervised MultiModal Versatile Networks (2020, background): “We build on top of [52, 78, 79] who learn good representations from such narrated material, but consider learning representations using audio as well.”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “uch as VQA and VCR, with great potential in extending to visual captioning [84, 70], visual dialog [47, 68], vision-language navigation [18, 43], as well as video-and-language representation learning [62, 61, 41, 31]. Recent work [7] also investigates the design of probing tasks to understand the kn”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “re is a growing interest in pre-training generic models to solve a variety of V+L problems, such as visual questionanswering (VQA), image-text retrieval and image captioning etc. The existing methods [37,38,22,5,46,35,19,10] employ BERT-like objectives [6] to learn crossmodal representations from a ”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, positive): “To avoid labeling visual data, several approaches have leveraged audio transcripts obtained from narrated videos using automatic speech recognition (ASR) as a way to supervise video models for object detection [4, 16, 56], captioning [34, 72], classification [3, 44, 49, 89], summarization [59] or re”
LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, methodology): “VideoBert (Sun et al., 2019) takes masked LM on the concatenation of language words and visual tokens, where the visual”
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “VideoBERT (Sun et al. 2019) is one such method, whose goal is to learn cross-modal representations from videos and their corresponding transcripts.”
Training data-efficient image transformers & distillation through attention (2020, background): “Nevertheless hybrid architectures that combine convnets and transformers, including the self-attention mechanism, have recently exhibited competitive results in image classification [53], detection [6, 26], video processing [42, 50], unsupervised object discovery [33], and unified text-vision tasks ”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Recent works have studied how to ground modern language models, by training them to accept additional environment inputs [30, 31, 32, 33, 34] or to directly output actions [35, 36, 37].”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “(Johnson et al., 2016), video-action captioning frameworks (Sun et al., 2019), or scene-graph parsers (Tang et al., 2020).”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”