Video-Text Pre-training with Learned Regions

By Rui Yan, Mike Zheng Shou, Yixiao Ge, Alex Wang, Xudong Lin, Guanyu Cai, Jinhui Tang. 2021

30 citations recorded by the source corpus.

Video-Text pre-training aims at learning transferable representations from large-scale video-text pairs via aligning the semantics between visual and textual information. State-of-the-art approaches extract visual features from raw pixels in an end-to-end fashion. However, these methods operate at frame-level directly and thus overlook the spatio-temporal structure of objects in video, which yet has a strong synergy with nouns in textual descriptions. In this work, we propose a simple yet effective module for video-text representation learning, namely RegionLearner, which can take into account the structure of objects during pre-training on large-scale video-text pairs. Given a video, our module (1) first quantizes continuous visual features via clustering patch-features into the same cluster according to content similarity, then (2) generates learnable masks to aggregate fragmentary features into regions with complete semantics, and finally (3) models the spatio-temporal dependencies between different semantic regions. In contrast to using off-the-shelf object detectors, our proposed module does not require explicit supervision and is much more computationally efficient. We pre-train the proposed approach on the public WebVid2M and CC3M datasets. Extensive evaluations on four downstream video-text retrieval benchmarks clearly demonstrate the effectiveness of our RegionLearner.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Rui Yan
2. Ying Shan
3. Alex Wang
4. Yixiao Ge
5. Mike Zheng Shou
6. Jianping Wu
7. Xiaohu Qie
8. Guanyu Cai

In Computer Vision and Pattern Recognition

1. Ying Shan
2. Yixiao Ge
3. Rui Yan
4. Alex Wang
5. Mike Zheng Shou
6. Jianping Wu
7. Xiaohu Qie
8. Guanyu Cai

In Multimodal Learning

1. Ying Shan
2. Rui Yan
3. Alex Wang
4. Yixiao Ge
5. Mike Zheng Shou
6. Xiaohu Qie
7. Guanyu Cai
8. Xudong Lin

In Representation & Self-Supervised Learning

1. Ying Shan
2. Rui Yan
3. Alex Wang
4. Yixiao Ge
5. Mike Zheng Shou
6. Xiaohu Qie
7. Xudong Lin
8. Yuying Ge

Citations in the corpus

• All in One: Exploring Unified Video-Language Pre-Training (2022, methodology): “Some recent methods [3, 26, 33, 55] have attempted to train models in an end-to-end fashion but still rely on well-trained visual encoders for feature extraction.”
• MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, unknown): “…years, consistently pushing the performance boundaries across a spectrum of downstream tasks (Li et al., 2020; Akbari et al., 2021; Fang et al., 2021; Yan et al., 2021; Li et al., 2021; Radford et al., 2021; Li et al., 2022; Zellers et al., 2022; Zeng et al., 2022b; Yang et al., 2022; Wang et…”
• VLP: A Survey on Vision-language Pre-training (2022, unknown): “ActBERT[135] Video OD-RFs+CNN Emb Single-stream No MLM+VLM+MVM HT100M AS+ASL+VC+VQ A+VLR”