ActBERT: Learning Global-Local Video-Text Representations

By Linchao Zhu, Yi Yang. 2020

468 citations recorded by the source corpus.

In this paper, we introduce ActBERT for self-supervised learning of joint video-text representations from unlabeled data. First, we leverage global action information to catalyze the mutual interactions between linguistic texts and local regional objects. It uncovers global and local visual clues from paired video sequences and text descriptions for detailed visual and text relation modeling. Second, we introduce an ENtangled Transformer block (ENT) to encode three sources of information, i.e., global actions, local regional objects, and linguistic descriptions. Global-local correspondences are discovered via judicious clues extraction from contextual information. It enforces the joint videotext representation to be aware of fine-grained objects as well as global human intention. We validate the generalization capability of ActBERT on downstream video-and language tasks, i.e., text-video clip retrieval, video captioning, video question answering, action segmentation, and action step localization. ActBERT significantly outperform the state-of-the-arts, demonstrating its superiority in video-text representation learning.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “Along the journey of VLP, researchers have investigated different training strategies [11,31], robustness [25], compression [9, 10, 42], probing analysis [3, 27], and the extension to video-text modeling [21, 24, 38, 39, 50].”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, background): “Later on, VidL pre-trained models [5, 15, 16, 46] built on the above pre-extracted features have shown promising results.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “Of late, works have sought to learn multimodal representations transferable to many tasks from uncurated sets of (usually how-to) videos [80, 106, 107, 81, 127, 9, 7, 4]; generally these are applied to video understanding tasks like activity recognition.”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “Given the advancements made in vision-language representation learning [41, 65], future works are likely better served by using pre-trained architectures.”
Unifying Vision-and-Language Tasks via Text Generation (2021, background): “…et al., 2020b; Cho et al., 2020; Radford et al., 2021; Zhang et al., 2021) and video+text pretraining models (Sun et al., 2019b;a; Li et al., 2020a; Zhu & Yang, 2020; Miech et al., 2020) have also shown to perform better than previous non-pretraining approaches (Yu et al., 2018a; Anderson et al.,…”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “Methods using such readily-available data have shown significant improvements on several tasks including video retrieval, action localization, action recognition and video captioning [25, 56, 57, 58, 71, 72, 100], sometimes outperforming fully-supervised baselines.”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, background): “, 2020), ActBERT (Zhu and Yang, 2020) and UniViLM (Luo et al.”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, methodology): “Table 8c shows CLIPBERT 8×2 improves ActBERT [83] model pre-trained on HowTo100M by 2.”
Support-set bottlenecks for video-text representation learning (2020, background): “A few works (Sun et al., 2019a;b; Zhu & Yang, 2020) extend the BERT model to accept both visual and textual tokens to learn high-level semantic video-text representations.”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “developed [57, 76, 45, 41, 26, 29, 36, 20, 50, 31, 58, 44, 39, 73] in this field are often evaluated on different tasks, datasets and experimental settings, making fair comparison difficult.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, background): “The other line of work adopts a single cross-modal encoder and concatenates the vision and text sequences as inputs, including VideoBERT (Sun et al., 2019b), Unicoder-VL (Li et al., 2020a), VL-BERT (Su et al., 2020), UNITER (Chen et al., 2020b), VLP (Zhou et al., 2018), ActBERT (Zhu and Yang, 2020) ”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, background): “, 2019), ActBERT (Zhu and Yang, 2020), NoiseE (Amrani et al.”
Compressing Visual-linguistic Model via Knowledge Distillation (2021, background): “Following the prominent progress in the transformer-based [68] pretraining in natural language [14, 51, 38, 5, 11, 52], visual-linguistic pre-training models, either for image+text [44, 64, 9, 42, 26, 81, 41, 18, 40, 45] or for video+text [61, 40, 46, 84, 39].”