HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training

By Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu, Qi Qian, Ji Zhang, Fei Huang. 2022

107 citations recorded by the source corpus.

Video-language pre-training has advanced the performance of various downstream video-language tasks. However, most previous methods directly inherit or adapt typical image-language pre-training paradigms to video-language pre-training, thus not fully exploiting the unique characteristic of video, i.e., temporal. In this paper, we propose a Hierarchical Temporal-Aware video-language pre-training framework, HiTeA, with two novel pre-training tasks for yielding temporal-aware multi-modal representation with cross-modal fine-grained temporal moment information and temporal contextual relations between video-text multi-modal pairs. First, we propose a cross-modal moment exploration task to explore moments in videos by mining the paired texts, which results in detailed video moment representation. Then, based on the learned detailed moment representations, the inherent temporal contextual relations are captured by aligning video-text pairs as a whole in different time resolutions with multi-modal temporal relation exploration task. Furthermore, we introduce the shuffling test to evaluate the temporal reliance of datasets and video-language pre-training models. We achieve state-of-the-art results on 15 well-established video-language understanding and generation tasks, especially on temporal-oriented datasets (e.g., SSv2-Template and SSv2-Label) with 8.6% and 11.1% improvement respectively. HiTeA also demonstrates strong generalization ability when directly transferred to downstream tasks in a zero-shot manner.

Subfields: Cancer Research, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Chenliang Li
2. Haiyang Xu
3. Ji Zhang
4. Anwen Hu
5. Hehong Chen
6. Ming Yan
7. Guohai Xu
8. Jiabo Ye

In Multimodal Learning

1. Chenliang Li
2. Haiyang Xu
3. Ming Yan
4. Guohai Xu
5. Ji Zhang
6. Qinghao Ye

Citations in the corpus

• mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality (2023, methodology): “In this paper, we present mPLUG-Owl with an innovative modularized training paradigm for large multi-modal language models that can support multiple modalities concurrently, drawing inspiration from the concept of modularization [Xu et al., 2023b, Li et al., 2022, Xu et al., 2021, Ye et al., 2022].”