Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning

By Werner Zellinger, Thomas Grubinger, Edwin David Lughofer, Thomas Natschläger, Susanne Saminger-Platz. 2017

675 citations recorded by the source corpus.

The learning of domain-invariant representations in the context of domain adaptation with neural networks is considered. We propose a new regularization method that minimizes the discrepancy between domain-specific latent feature representations directly in the hidden activation space. Although some standard distribution matching approaches exist that can be interpreted as the matching of weighted sums of moments, e.g. Maximum Mean Discrepancy (MMD), an explicit order-wise matching of higher order moments has not been considered before. We propose to match the higher order central moments of probability distributions by means of order-wise moment differences. Our model does not require computationally expensive distance and kernel matrix computations. We utilize the equivalent representation of probability distributions by moment sequences to define a new distance function, called Central Moment Discrepancy (CMD). We prove that CMD is a metric on the set of probability distributions on a compact interval. We further prove that convergence of probability distributions on compact intervals w.r.t. the new metric implies convergence in distribution of the respective random variables. We test our approach on two different benchmark data sets for object recognition (Office) and sentiment analysis of product reviews (Amazon reviews). CMD achieves a new state-of-the-art performance on most domain adaptation tasks of Office and outperforms networks trained with MMD, Variational Fair Autoencoders and Domain Adversarial Neural Networks on Amazon reviews. In addition, a post-hoc parameter sensitivity analysis shows that the new approach is stable w.r.t. parameter changes in a certain interval. The source code of the experiments is publicly available.

Subfields: Artificial Intelligence, Cancer Research, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiashi Feng
2. Jian Liang
3. Kate Saenko
4. Dacheng Tao
5. Mingsheng Long
6. Dapeng Hu
7. Trevor Darrell
8. Tao A. Xiang

In Cancer Research

1. Kate Saenko
2. Jian Liang
3. Jiashi Feng
4. Xingchao Peng
5. Zhangjie Cao
6. Tao A. Xiang
7. Jindong Wang
8. Dapeng Hu

In Representation & Self-Supervised Learning

1. Jian Liang
2. Mingsheng Long
3. Jianmin Wang
4. Jiashi Feng
5. Kate Saenko
6. Zhangjie Cao
7. Dapeng Hu
8. Tao A. Xiang

Citations in the corpus

• Learning to cluster in order to Transfer across domains and tasks (2017, background): “…of work dealing with domain shift between image datasets by minimizing domain discrepancy (Tzeng et al., 2014; Long et al., 2015b; Ganin et al., 2016; Sun et al., 2016; Long et al., 2016; Sener et al., 2016; Carlucci et al., 2017; Long et al., 2017; Zellinger et al., 2017; Bousmalis et al., 2017).”
• Learngene: From Open-World to Your Learning Task (2021, background): “Transfer learning: Methods such as [26, 59, 60, 53, 36, 71, 15, 74, 16], which focus on transferring the knowledge across domains, aim at improving the performance of target learners in target domains.”
• Moment Matching for Multi-Source Domain Adaptation (2018, background): “moment matching [47], adversarial domain confusion [40, 8, 38] and GAN-based alignment [50, 15, 23].”
• MMD GAN: Towards Deeper Understanding of Moment Matching Network (2017, neutral): “However, the proposed algorithm requires matrix (tensor) decompositions because of exact moment matching [24], which is hard to scale to higher order moment matching.”
• A Comprehensive Survey on Transfer Learning (2019, background): “2, there are some other measurement criteria adopted in transfer learning, including Wasserstein distance [59], [60], central moment discrepancy [61], etc.”
• Controllable Invariance through Adversarial Feature Learning (2017, methodology): “, 2016] and central moment discrepancy (CMD) [Zellinger et al., 2017] have been utilized to learn features that are domain invariant.”
• No More Discrimination: Cross City Adaptation of Road Scene Segmenters (2017, methodology): “[40] consider Central Moment Discrepancy (CMD) instead of MMD, while Sener et al.”
• Cross-Domain Gradient Discrepancy Minimization for Unsupervised Domain Adaptation (2021, methodology): “In addition, central moment discrepancy (CMD) [42] and maximum density divergence (MDD) [16] are another two criterias to align feature distributions in hidden layers.”
• Stochastic Classifiers for Unsupervised Domain Adaptation (2020, methodology): “For GA, many metrics, including Maximum Mean Discrepancy (MMD) [28, 29], Central Moment Discrepancy (CMD) [60] and Wasserstein distance [47], have been proposed or utilised.”
• Importance Weighted Adversarial Nets for Partial Domain Adaptation (2018, methodology): “maximum mean discrepancy (MMD) [25, 14, 18, 17], Central Moment Discrepancy (CMD) [27], and second-order statistics matching [22].”
• MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (2020, methodology): “Amongst many choices, we use the Central Moment Discrepancy (CMD) [63] metric for this purpose.”
• A Balanced and Uncertainty-aware Approach for Partial Domain Adaptation (2020, methodology): “Typically, existing UDA methods minimize the discrepancy between two domains by matching their statistical distribution moments [41,37,47,18] or by domain adversarial learning [39,10,40,26].”
• Domain-Specific Batch Normalization for Unsupervised Domain Adaptation (2019, methodology): “Global alignment techniques include Maximum Mean Discrepancy (MMD) [1, 28], Central Moment Discrepancy (CMD) [29], Wassterstein distance [21], and CORrelation ALignment (CORAL) [22], some of which are reformulated using deep neural networks [12, 23, 28].”
• Temporal Attentive Alignment for Large-Scale Video Domain Adaptation (2019, methodology): “To measure the alignment between different domains, we use Maximum Mean Discrepancy (MMD) and domain loss, which are calculated using the final video representations.”
• Deep Subdomain Adaptation Network for Image Classification (2020, neutral): “The first is statistic moment matching based approach, i.e. maximum mean discrepancy (MMD) [13], [26], [27], Central Moment Discrepancy (CMD) [28], and second-order statistics matching [16].”
• Generative Adversarial Networks (GANs) (2020, methodology): “As a solution [143][144], it is suggested to use an alternative family of distance functions, IPMs which includes Wasserstein distance and RKHS-induced maximum mean discrepancies.”