Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training

By Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, Jingjing Liu. 2020

565 citations recorded by the source corpus.

We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer via multimodal fusion, and global video context is captured by a Temporal Transformer. In addition to standard Masked Language Modeling (MLM) and Masked Frame Modeling (MFM) objectives, we design two new pre-training tasks: (i) Video-Subtitle Matching (VSM), where the model predicts both global and local temporal alignment; and (ii) Frame Order Modeling (FOM), where the model predicts the right order of shuffled video frames. HERO is jointly trained on HowTo100M and large-scale TV datasets to gain deep understanding of complex social dynamics with multi-character interactions. Comprehensive experiments demonstrate that HERO achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains. We also introduce two new challenging benchmarks How2QA and How2R for Video QA and Retrieval, collected from diverse video content over multimodalities.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “tion [16, 40], as well as video-and-language representation learning [56, 55, 38, 29].”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “Table 6 shows the performance of the representative approaches used in Table 2 on TVQA, How2QA, TVC, and YC2C.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “Along the journey of VLP, researchers have investigated different training strategies [11,31], robustness [25], compression [9, 10, 42], probing analysis [3, 27], and the extension to video-text modeling [21, 24, 38, 39, 50].”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “Later on, VidL pre-trained models [5, 15, 16, 46] built on the above pre-extracted features have shown promising results.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, positive): “As result, current VideoQA datasets [11, 16, 17, 21, 27, 33, 38, 43, 46, 60, 69, 74, 84, 90, 91, 92, 93] are limited in size, as the largest, TGIF-QA [33], contains only 72K annotated clips (see Figure 2 for more details).”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “During pre-training, Masked Language Modeling (MLM) and Image-Text Matching (ITM) tasks have been widely used (Wang et al., 2020; Fang et al., 2021c; Li et al., 2020; Zhang et al., 2021a; Chen et al., 2020b; Dou et al., 2021; Wang et al., 2021a; Kim et al., 2021).”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, methodology): “In Table 7a, CLIPBERT achieves significant performance gain over existing methods on MSRVTT retrieval, including HT [46], ActBERT [83], and HERO [37], which are pre-trained on 136M clip-caption pairs from HowTo100M [46].”
Is Space-Time Attention All You Need for Video Understanding? (2021, methodology): “Finally, multimodal video-text transformers (Sun et al., 2019; Li et al., 2020a) have also been trained or pretrained in unsupervised fashion by adopting masked-token pretext tasks adapted from the language domain (Devlin et al., 2018; Radford et al., 2018).”
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models (2020, background): “CBT [29] introduced contrastive learning to handle real-valued video frame features, and HERO [20] proposed hierarchical Transformer architectures to leverage both global and local temporal visual-textual alignments.”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, methodology): “To construct a comprehensive evaluation benchmark, we include recent datasets collected on multi-channel videos: TVR [35], How2R [36], TVQA [32], How2QA [36], VIOLIN [40], VLEP [34] and TVC [35].”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, methodology): “CBT (Sun et al., 2019a), HERO (Li et al., 2020b), VideoAsMT (Korbar et al., 2020) and UniVL (Luo et al., 2020) adopt multi-task learning (MTL) for pre-training on retrieval tasks.”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, background): “, 2020), HERO (Li et al., 2020), ClipBERT (Lei et al.”
Compressing Visual-linguistic Model via Knowledge Distillation (2021, background): “Following the prominent progress in the transformer-based [68] pretraining in natural language [14, 51, 38, 5, 11, 52], visual-linguistic pre-training models, either for image+text [44, 64, 9, 42, 26, 81, 41, 18, 40, 45] or for video+text [61, 40, 46, 84, 39].”
Look Before you Speak: Visually Contextualized Utterances (2020, methodology): “How2QA: How2QA [47] consists of QA annotations for the HowTo100M dataset.”
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners (2022, background): “A particular limitation of most recent video-language pretraining frameworks [28, 21, 60, 67, 66, 25, 63] is that they are encoder-only, which means they do not have the ability to generate text from videos for purposes such as captioning [61, 56], question answering [59], and future prediction [23]”