Egocentric Video-Language Pretraining

By Kevin Qinghong Lin, Alex Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z. Xu, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong and 5 more. 2022

295 citations recorded by the source corpus.

Video-Language Pretraining (VLP), which aims to learn transferable representation to advance a wide range of video-text downstream tasks, has recently received increasing attention. Best performing works rely on large-scale, 3rd-person video-text datasets, such as HowTo100M. In this work, we exploit the recently released Ego4D dataset to pioneer Egocentric VLP along three directions. (i) We create EgoClip, a 1st-person video-text pretraining dataset comprising 3.8M clip-text pairs well-chosen from Ego4D, covering a large variety of human daily activities. (ii) We propose a novel pretraining objective, dubbed EgoNCE, which adapts video-text contrastive learning to the egocentric domain by mining egocentric-aware positive and negative samples. (iii) We introduce EgoMCQ, a development benchmark that is close to EgoClip and hence can support effective validation and fast exploration of our design decisions in EgoClip and EgoNCE. Furthermore, we demonstrate strong performance on five egocentric downstream tasks across three datasets: video-text retrieval on EPIC-KITCHENS-100; action recognition on Charades-Ego; natural language query, moment query, and object state change classification on Ego4D challenge benchmarks. The dataset and code are available at https://github.com/showlab/EgoVLP.

Subfields: Cancer Research, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cancer Research

1. Yu Qiao

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Cordelia Schmid
3. Andrew Zisserman
4. Yu Qiao
5. Xiaogang Wang
6. Jitendra Malik
7. Dacheng Tao
8. Trevor Darrell

In Multimodal Learning

1. Cordelia Schmid
2. Arsha Nagrani
3. Josef Sivic
4. Ivan Laptev
5. Paul Hongsuck Seo
6. Antoine Miech
7. Yixiao Ge
8. Alex Wang

Citations in the corpus

• All in One: Exploring Unified Video-Language Pre-Training (2022, methodology): “Some recent methods [3, 26, 33, 55] have attempted to train models in an end-to-end fashion but still rely on well-trained visual encoders for feature extraction.”
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “[64] Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, et al.”
• Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “This data is valuable for mining for taxonomies of objects and actions in the data, indexing the videos with keywords for exploring the dataset, and for future research in video-language learning, as has been quite successful for the Ego4D narrations [8, 85, 123].”
• EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, methodology): “To generate pairs of captions, embodied plannings, and corresponding video segments with time intervals, we utilized the EgoVLP framework [54] to segment the video.”