CLIP4Caption ++: Multi-CLIP for Video Caption

By Mingkang Tang, Zhanyu Wang, Zhaoyang Zeng, Feng Rao, Dian Li. 2021

8 citations recorded by the source corpus.

This report describes our solution to the VALUE Challenge 2021 in the captioning task. Our solution, named CLIP4Caption++, is built on X-Linear/X-Transformer, which is an advanced model with encoder-decoder architecture. We make the following improvements on the proposed CLIP4Caption++: We employ an advanced encoder-decoder model architecture X-Transformer as our main framework and make the following improvements: 1) we utilize three strong pre-trained CLIP models to extract the text-related appearance visual features. 2) we adopt the TSN sampling strategy for data enhancement. 3) we involve the video subtitle information to provide richer semantic information. 3) we introduce the subtitle information, which fuses with the visual features as guidance. 4) we design word-level and sentence-level ensemble strategies. Our proposed method achieves 86.5, 148.4, 64.5 CIDEr scores on VATEX, YC2C, and TVC datasets, respectively, which shows the superior performance of our proposed CLIP4Caption++ on all three datasets.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Jianfeng Wang
5. Linjie Li
6. Luc Van Gool
7. Xiaowei Hu
8. Cordelia Schmid

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Jianfeng Wang
5. Linjie Li
6. Xiaowei Hu
7. Zhengyuan Yang
8. Ce Liu

Citations in the corpus

• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “5) than CLIP4Caption++ (Tang et al., 2021), which relies on model ensemble and additional subtitle input.”