TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
By Jianwei Yang, Yonatan Bisk, Jianfeng Gao. 2021
161 citations recorded by the source corpus.
Contrastive learning has been widely used to train transformer-based vision-language models for video-text alignment and multi-modal representation learning. This paper presents a new algorithm called Token-Aware Cascade contrastive learning (TACo) that improves contrastive learning using two novel techniques. The first is the token-aware contrastive loss which is computed by taking into account the syntactic classes of words. This is motivated by the observation that for a video-text pair, the content words in the text, such as nouns and verbs, are more likely to be aligned with the visual contents in the video than the function words. Second, a cascade sampling method is applied to generate a small set of hard negative examples for efficient loss estimation for multi-modal fusion layers. To validate the effectiveness of TACo, in our experiments we finetune pretrained models for a set of downstream tasks including text-video retrieval (YouCook2, MSR-VTT and ActivityNet), video action step localization (CrossTask), video action segmentation (COIN). The results show that our models attain consistent improvements across different experimental settings over previous methods, set-ting new state-of-the-art on three public text-video retrieval benchmarks of YouCook2, MSR-VTT and ActivityNet.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Cordelia Schmid
- Andrew Zisserman
- Ivan Laptev
- Josef Sivic
- Arsha Nagrani
- Antoine Miech
- Ping Luo
- Mohit Bansal
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Cordelia Schmid
- Andrew Zisserman
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Andrea Vedaldi
In Multimodal Learning
- Cordelia Schmid
- Arsha Nagrani
- Josef Sivic
- Ivan Laptev
- Paul Hongsuck Seo
- Antoine Miech
- Yixiao Ge
- Alex Wang
Citations in the corpus
- Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “[116] Jianwei Yang, Yonatan Bisk, and Jianfeng Gao.”
- MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval (2022, background): “0 TACo [53] 2021 I3D, S3D HowTo100M 120M 28.”
- CLIP2TV: An Empirical Study on Transformer-based Methods for Video-Text Retrieval (2021, methodology): “Same as TACo [17] and AL-BEF [8], our multi-modal encoder consists of self-attention layers.”
- X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval (2022, background): “Moreover, TACo [57] introduces token-level and sentence-level loss to consider both fine-grained and coarse-grained contrast.”
- BridgeFormer: Bridging Video-text Retrieval with Multiple Choice Questions (2022, background): “Methods in the first category [6, 11, 13, 22, 25, 28, 33, 41, 43] adopt two individual encoders to embed video features and text features, and project them into the same latent space.”
- Multimodal Learning With Transformers: A Survey (2022, background): “[119], [260], [261], [262] has led to a surge of leveraging large quantities of web data (e.”
- Language Models Can See: Plugging Visual Controls in Text Generation (2022, background): “Xu et al. [84] and Yang et al. [86] propose a contrastive pre-training approach for video-text alignment.”