Multi-Modal Domain Adaptation for Fine-Grained Action Recognition
By Jonathan Munro, Dima Damen. 2019
85 citations recorded by the source corpus.
Fine-grained action recognition datasets exhibit environmental bias, where even the largest datasets contain sequences from a limited number of environments due to the challenges of large-scale data collection. We show that multi-modal action recognition models suffer with changes in environment, due to the differing levels of robustness of each modality. Inspired by successes in adversarial training for unsupervised domain adaptation, we propose a multi-modal approach for adapting action recognition models to novel environments. We employ late fusion of the two modalities commonly used in action recognition (RGB and Flow), with multiple domain discriminators, so alignment of modalities is jointly optimised with recognition. We test our approach on EPIC Kitchens, proposing the first benchmark for domain adaptation of fine-grained actions. Our multi-modal method outperforms single-modality alignment as well as other alignment methods by up to 3%.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Biswajeet Pradhan
- Manoranjan Paul
- Subrata Chakraborty
- Michael James Horry
- Ying Sun
- Yi Cheng
- Fen Fang
- Nagesh Shukla
Citations in the corpus
- COVID-19 Detection Through Transfer Learning Using Multimodal Imaging Data (2020, methodology): “Although data fusion comes with its own set of challenges [97], [98], it has been used successfully in other application areas such as remote sensing [99]–[101], action detection [102], and medical diagnosis and imaging [103], [104].”
- Team VI-I2R Technical Report on EPIC-KITCHENS-100 Unsupervised Domain Adaptation Challenge for Action Recognition 2021 (2022, background): “To the best of our knowledge, MM-SADA [6] is the first attempt on UDA for fine-grained action recognition.”
- Fine-Grained Image Analysis With Deep Learning: A Survey (2021, unknown): “More research is needed in areas such as domain adaptation [207], [208], [209], long-tailed distributions [210], [211], open world settings [212], scale variations [2], fine-grained video understanding [213], [214], knowledge transfer, and resource constrained embedded deployment, to name a few.”