A Unified Framework for Multimodal Domain Adaptation

By Fan Qi, Xiaoshan Yang, Changsheng Xu. 2018

69 citations recorded by the source corpus.

Domain adaptation aims to train a model on labeled data from a source domain while minimizing test error on a target domain. Most of existing domain adaptation methods only focus on reducing domain shift of single-modal data. In this paper, we consider a new problem of multimodal domain adaptation and propose a unified framework to solve it. The proposed multimodal domain adaptation neural networks(MDANN) consist of three important modules. (1) A covariant multimodal attention is designed to learn a common feature representation for multiple modalities. (2) A fusion module adaptively fuses attended features of different modalities. (3) Hybrid domain constraints are proposed to comprehensively learn domain-invariant features by constraining single modal features, fused features, and attention scores. Through jointly attending and fusing under an adversarial objective, the most discriminative and domain-adaptive parts of the features are adaptively fused together. Extensive experimental results on two real-world cross-domain applications (emotion recognition and cross-media retrieval) demonstrate the effectiveness of the proposed method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “*: Note that Test* refers to ‘Target Test’ Dataset Train Test Classes SD M AAL Year Real/Syn Im a g e Office [105] 4110 N/A 31 1 3 N/A 2010 Real ImageCLEF [106] 2400 600 12 5 1 N/A 2014 Real Office↔ Home [107] 15500 N/A 65 4 1 N/A 2017 Real VisDA-C [108] 280157 N/A 12 3 1 N/A 2017 Real/Syn DomainNet”
Multi-Modal Domain Adaptation for Fine-Grained Action Recognition (2019, background): “Utilising multiple modalities (image and audio) for UDA has been recently investigated for bird image retrieval [39].”
Multi-Modal Domain Adaptation for Fine-Grained Action Recognition (2020, background): “Utilising multiple modalities (image and audio) for UDA has been recently investigated for bird image retrieval [39].”