Multi-Modal Domain Adaptation for Fine-Grained Action Recognition

By Jonathan Munro, Dima Damen. 2020

238 citations recorded by the source corpus.

Fine-grained action recognition datasets exhibit environmental bias, where multiple video sequences are captured from a limited number of environments. Training a model in one environment and deploying in another results in a drop in performance due to an unavoidable domain shift. Unsupervised Domain Adaptation (UDA) approaches have frequently utilised adversarial training between the source and target domains. However, these approaches have not explored the multi-modal nature of video within each domain. In this work we exploit the correspondence of modalities as a self-supervised alignment approach for UDA in addition to adversarial alignment (Fig. 1). We test our approach on three kitchens from the large-scale EPIC-Kitchens dataset, using two modalities commonly employed for action recognition: RGB and Optical Flow. We show that multi-modal self-supervision alone improves the performance over source-only training by 2.4% on average. We then combine adversarial training with multi-modal self-supervision, showing that our approach outperforms other UDA methods by 3%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “Table 8 shows EPIC-KITCHENS-100 provides several advantages over videobased datasets: largest number of instances, classes, subdomains, and is multimodal [94].”
COVID-19 Detection Through Transfer Learning Using Multimodal Imaging Data (2020, methodology): “Although data fusion comes with its own set of challenges [97], [98], it has been used successfully in other application areas such as remote sensing [99]–[101], action detection [102], and medical diagnosis and imaging [103], [104].”
Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment (2021, methodology): “We validate or method in the DA context against four existing unsupervised domain adaptation approaches: (a) AdaBN [32]: Batch Normalization layers are updated with target domain statistics; (b) MMD [36]: it minimizes separate discrepancy measures applied to single modalities; (c) Adversarial Only [”
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines (2020, methodology): “Of these, recent research has explored using EPIC-KITCHENS for: video object reasoning and detection [67], [68], action retrieval [69], visual learning of novel words [57], unsupervised domain adaptation [70] and learning environmental affordances [71].”
EPIC-KITCHENS-100 Unsupervised Domain Adaptation Challenge for Action Recognition 2021: Team M3EM Technical Report (2021, unknown): “Previous works [6, 8] have shown that using multiple modalities can improve the performance of UDA on action recognition, but none of them considered using early fusion on the modalities to enhance the transferability of the generated features.”
Team VI-I2R Technical Report on EPIC-KITCHENS-100 Unsupervised Domain Adaptation Challenge for Action Recognition 2021 (2022, background): “To the best of our knowledge, MM-SADA [6] is the ﬁrst attempt on UDA for ﬁne-grained action recognition.”