VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation

By Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, R. Radhakrishna Pillai, Yu Cheng, Luowei Zhou, X. Wang, William Yang Wang and 5 more. 2021

122 citations recorded by the source corpus.

Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task. In reality, a truly useful VidL system is expected to be easily generalizable to diverse tasks, domains, and datasets. To facilitate the evaluation of such systems, we introduce Video-And-Language Understanding Evaluation (VALUE) benchmark, an assemblage of 11 VidL datasets over 3 popular tasks: (i) text-to-video retrieval; (ii) video question answering; and (iii) video captioning. VALUE benchmark aims to cover a broad range of video genres, video lengths, data volumes, and task difficulty levels. Rather than focusing on single-channel videos with visual information only, VALUE promotes models that leverage information from both video frames and their associated subtitles, as well as models that share knowledge across multiple tasks. We evaluate various baseline methods with and without large-scale VidL pre-training, and systematically investigate the impact of video input channels, fusion methods, and different video representations. We also study the transferability between tasks, and conduct multi-task learning under different settings. The significant gap between our best model and human performance calls for future study for advanced VidL models. VALUE is available at https://value-benchmark.github.io/.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Jianfeng Gao
3. Mohit Bansal
4. Lijuan Wang
5. Zicheng Liu
6. Jianfeng Wang
7. Linjie Li
8. Xiaowei Hu

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Cordelia Schmid
3. Dacheng Tao
4. Trevor Darrell
5. Jitendra Malik
6. Zhe Gan
7. Ming-Hsuan Yang
8. Xiaogang Wang

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Jianfeng Wang
5. Linjie Li
6. Xiaowei Hu
7. Zhengyuan Yang
8. Mohit Bansal

In Evaluation & Benchmarks

1. Mohit Bansal
2. Fei Sha
3. Zhe Gan
4. Zicheng Liu
5. Lijuan Wang
6. Jiajun Wu
7. Jiahui Yu
8. Juan Carlos Niebles

Citations in the corpus

• VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “As for video-text experiments, we apply our method on four tasks from VALUE [27] benchmark: TVQA [22] and How2QA [26] for video question answering, TVC [23] and YC2C [56] for video captioning.”
• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[64] Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, et al.”
• Multimodal Learning With Transformers: A Survey (2022, background): “MULTIMODALQA (MMQA) [131], VALUE [132], Fashion IQ [133], LRS2-BBC [134], ActivityNet [135], VisDial [136].”
• Co-training Transformer with Videos and Images Improves Action Recognition (2021, methodology): “from improved action recognition modeling, such as video retrieval [12, 17], video captioning [17], video QA [18], etc.”
• Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “Exceptions are How2QA and TVQA for which the authors [52, 61] provide speech for all videos.”
• DoRA: Weight-Decomposed Low-Rank Adaptation (2024, background): “Moreover, DoRA achieves roughly 2% higher accuracy than LoRA in video-text understanding tasks.”
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, unknown): “For video MLLMs, similar efforts have been made to incorporate existing benchmarks [29, 58] for evaluating video understanding [27, 47].”
• SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning (2021, methodology): “Our method achieves better performance than VALUE [7].”
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, unknown): “Theyfightover the ball, tryingto getitintothe goal.”
• Revisiting the “Video” in Video-Language Understanding (2022, methodology): “…the same preprocessing as our model, and all models leverage the same CLIP features (HERO baselines additionally leverage heavy motion features [11,30]). racies are competitive with state-of-the-art video models, even with relatively small n sample sizes, suggesting the promise for analyzing…”
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, background): “A longer-term outlook concerns including other tasks and languages into the benchmark, as well as reaching beyond the image-only visual modality, towards multimodal tasks that rely on processing videos (Li et al., 2021) and speech (Yang et al., 2021) in different languages.”