Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling

By Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, Jingjing Liu. 2021

804 citations recorded by the source corpus.

The canonical approach to video-and-language learning (e.g., video question answering) dictates a neural model to learn from offline-extracted dense video features from vision models and text features from language models. These feature extractors are trained independently and usually on tasks different from the target domains, rendering these fixed features sub-optimal for downstream tasks. Moreover, due to the high computational overload of dense video features, it is often difficult (or infeasible) to plug feature extractors directly into existing approaches for easy finetuning. To provide a remedy to this dilemma, we propose a generic framework CLIPBERT that enables affordable endto-end learning for video-and-language tasks, by employing sparse sampling, where only a single or a few sparsely sampled short clips from a video are used at each training step. Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that CLIPBERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end-to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full-length videos, proving the proverbial less-is-more principle. Videos in the datasets are from considerably different domains and lengths, ranging from 3-second genericdomain GIF videos to 180-second YouTube human activity videos, showing the generalization ability of our approach. Comprehensive ablation studies and thorough analyses are provided to dissect what factors lead to this success. Our code is publicly available.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “ClipBERT [23] is the recent state-of-the-art framework that enables end-to-end learning for video-and-language tasks including video question answering by employing sparse sampling.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “Of note, it nearly doubles [122]’s SoTA zero-shot accuracy on MSR-VTT QA, and it outperforms supervised approaches (like ClipBERT [73]) on STAR.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “Along the journey of VLP, researchers have investigated different training strategies [11,31], robustness [25], compression [9, 10, 42], probing analysis [3, 27], and the extension to video-text modeling [21, 24, 38, 39, 50].”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “[56]’s ClipBERT model learns vision-language representations from image captions, which more literally describe image content versus the longer ASR transcripts we consider.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “video domain (VideoQA), spatio-temporal video representations in terms of motion and appearance have been used in [22, 26, 31, 33, 34, 35, 40, 41, 42, 48, 84, 86, 94, 102].”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[58] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L.”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “Recent work [31, 44, 9] show that with image-text pre-training, models trained on 2D features alone can achieve decent performance on many video-and-language (VidL) tasks.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, methodology): “…most supervised DiDeMo dataset R@1 ↑ R@5 S UPERVISED S2VT (Venugopalan et al., 2014) 11.9 33.6 FSE (Zhang et al., 2018) 13.9 44.5 CE (Liu et al., 2019a) 16.1 41.1 ClipBERT (Lei et al., 2021) 20.4 48.0 Z ERO - SHOT VideoCLIP (Zero-shot) 16.6 46.9 Table 2: Text → video retrieval on DiDeMo.”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, methodology): “Furthermore, ClipBERT (Lei et al., 2021) proposed an efficient end-to-end approach through sparse sampling and presented that the pretrained by image-language dataset facilitated a better initialization of video-text retrieval.”
Compressing Visual-linguistic Model via Knowledge Distillation (2021, background): “Following the prominent progress in the transformer-based [68] pretraining in natural language [14, 51, 38, 5, 11, 52], visual-linguistic pre-training models, either for image+text [44, 64, 9, 42, 26, 81, 41, 18, 40, 45] or for video+text [61, 40, 46, 84, 39].”
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners (2022, methodology): “A particular limitation of most recent video-language pretraining frameworks [28, 21, 60, 67, 66, 25, 63] is that they are encoder-only, which means they do not have the ability to generate text from videos for purposes such as captioning [61, 56], question answering [59], and future prediction [23]”
Video Graph Transformer for Video Question Answering (2022, methodology): “Cross-modal pretraining seems promising [29,67,70].”
X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval (2022, methodology): “Recent years have witnessed the rapid development of VTR, which is supported by a series of pre-training multi-modal models [4, 30, 44], innovative retrieval methods [3, 5, 13–15, 24, 30, 34, 35, 38, 41, 54, 58, 61, 63, 66] and video-text benchmarks [2, 6, 7, 45, 56].”
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization (2021, background): “…Chen et al., 2020; Su et al., 2019; Cho et al., 2021; Sun et al., 2019; Li et al., 2020c; Zhu and Yang, 2020; Miech et al., 2020; Li et al., 2020b; Lei et al., 2021) on large-scale image-text corpus (Sharma et al., 2018; Chen et al., 2015; Kr-ishna et al., 2017) and video-text corpus (Lei et al.,…”
BridgeFormer: Bridging Video-text Retrieval with Multiple Choice Questions (2022, background): ““Joint-encoder” methods [20, 21, 23, 38, 40, 44] (see Fig.”
Playing Lottery Tickets with Vision and Language (2021, neutral): “This pretrain-thenfinetune learning paradigm has also been extended to diverse V+L tasks such as novel object captioning [26], visual dialog [51, 71], vision-language navigation [24, 49, 25], OCR-based VQA [73], robust VQA [38], and video-text pre-training [64, 63, 37, 82, 35].”