Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos
By Yanghao Li, Tushar Nagarajan, Bo Xiong, Kristen Grauman. 2021
127 citations recorded by the source corpus.
We introduce an approach for pre-training egocentric video models using large-scale third-person video datasets. Learning from purely egocentric data is limited by low dataset scale and diversity, while using purely exocentric (third-person) data introduces a large domain mismatch. Our idea is to discover latent signals in third-person video that are predictive of key egocentric-specific properties. Incorporating these signals as knowledge distillation losses during pre-training results in models that benefit from both the scale and diversity of third-person video data, as well as representations that capture salient egocentric properties. Our experiments show that our "Ego-Exo" framework can be seamlessly integrated into standard video models; it outperforms all baselines when fine-tuned for egocentric activity recognition, achieving state-of-the-art results on Charades-Ego and EPIC-Kitchens-100.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Kristen Grauman
- Luc Van Gool
- Jitendra Malik
- Trevor Darrell
- Andrew Zisserman
- Dacheng Tao
- Cordelia Schmid
- Ming-Hsuan Yang
In Multimodal Learning
- Kristen Grauman
- Lorenzo Torresani
- Dhruv Batra
- Antonio Torralba
- Rohit Girdhar
- Christoph Feichtenhofer
- Satwik Kottur
- Devi Parikh
Citations in the corpus
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “Notably, this domain shift makes it difficult for data-driven egocentric models to benefit from the standard paradigm of pretraining on third person Internet data (Li et al. 2021b; Sigurdsson et al. 2018).”
- Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Notably, pre-training egocentric video models with third-person data [70,221,224,239] suffers from the sizeable domain mismatch [139, 201].”
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “Prior work considers cross-view learning with unpaired videos [7, 82, 182] and view-invariant feature learning on paired videos [144].”
- A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, background): “[233] generated a model that exploited knowledge distillation loss during pre-training to obtain both the scale and diversity of third-person video data, as well as representations with prominent egocentric properties.”