Marginalized Denoising Autoencoders for Domain Adaptation

By Minmin Chen, Zhixiang Eddie Xu, Kilian Q. Weinberger, Fei Sha. 2012

838 citations recorded by the source corpus.

Stacked denoising autoencoders (SDAs) have been successfully used to learn new representations for domain adaptation. Recently, they have attained record accuracy on standard benchmark tasks of sentiment analysis across different text domains. SDAs learn robust data representations by reconstruction, recovering original features from data that are artificially corrupted with noise. In this paper, we propose marginalized SDA (mSDA) that addresses two crucial limitations of SDAs: high computational cost and lack of scalability to high-dimensional features. In contrast to SDAs, our approach of mSDA marginalizes noise and thus does not require stochastic gradient descent or other optimization algorithms to learn parameters--in fact, they are computed in closed-form. Consequently, mSDA, which can be implemented in only 20 lines of MATLAB™, significantly speeds up SDAs by two orders of magnitude. Furthermore, the representations learnt by mSDA are as effective as the traditional SDAs, attaining almost identical accuracies in benchmark tasks.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Representation Learning: A Review and New Perspectives (2012, methodology): “In the related domain adaptation setup, the target remains the same but the input distribution changes (Glorot et al., 2011b; Chen et al., 2012).”
Domain-Adversarial Training of Neural Networks (2015, methodology): “On one hand, we reproduce the results of Chen et al. (2012), which noticed that the mSDA representations have greater PAD values than original (raw) data.”
Unsupervised Cross-Domain Image Generation (2016, background): “Unsupervised domain adaptation addresses the following problem: given a labeled training set in S × Y , for some target space Y , and an unlabeled set of samples from domain T , learn a function h : T → Y (Chen et al., 2012; Ganin et al., 2016).”
Learning Transferable Features with Deep Adaptation Networks (2015, background): “Hence, deep neural networks have been explored for domain adaptation (Glorot et al., 2011; Chen et al., 2012), multimodal and multi-source learning problems (Ngiam et al., 2011; Ge et al., 2013), where signiﬁcant performance gains have been witnessed.”
Collaborative Deep Learning for Recommender Systems (2014, background): “If we assume that both the clean input Xc and the corrupted input X0 are observed, similar to [4, 19, 3, 7], we can define the following generative process:”
Unsupervised Adversarial Invariance (2018, methodology): “ent literature [6, 14] where the goal is to make the prediction task invariant to the “domain” information. We evaluate the performance of our model at domain adaptation on the Amazon Reviews dataset [4] using the same preprocessing as [14]. The dataset contains text reviews on products in four doma”
The Variational Fair Autoencoder (2015, methodology): “This was performed in an unsupervised way with mSDA (Chen et al., 2012) by training denoising autoencoders jointly on all domains, thus implicitly obtaining a representation general enough to explain both the domain and the data.”
Domain Generalization for Object Recognition with Multi-task Autoencoders (2015, methodology): “Some feature learning methods have been successfully applied to domain adaptation or transfer learning applications [9, 13].”
A Comprehensive Survey on Transfer Learning (2019, methodology): “The solution of W is given by [98], [99]”
Domain-Adversarial Neural Networks (2014, methodology): “On one hand, we reproduce the results of Chen et al. (2012), which noticed that the mSDA representations gave greater PAD values than those obtained with the original (raw) data.”
Causal Transfer in Machine Learning (2015, methodology): “Chen et al. [2012] proposes to build shared feature mappings which are robust to noise by using autoencoders.”
Unified Deep Supervised Domain Adaptation and Generalization (2017, background): “We compared our results with three UDA (GFK [26], mSDA [8], and RTML [13]) and one SDA (CDML [64]) algorithms under the same settings.”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, methodology): “ing autoencoders. This is done by training a multi-layer neural network to reconstruct input data from partial random corruptions with backpropagation. The Stacked Marginalized Denoising Autoencoders [12] 28 This is done by replacing the class prediction layer to correspond to the new set of classes”
A Survey on Metric Learning for Feature Vectors and Structured Data (2013, background): “Other relevant criteria should be investigated, for instance learning a metric that is robust to noise or invariant to some transformations of interest, in the spirit of denoising autoencoders (Vincent et al., 2008; Chen et al., 2012).”
Polyglot: Distributed Word Representations for Multilingual NLP (2013, background): “These embeddings are generated as a result of training “deep” architectures, and it has been shown that such representations are well suited for domain adaptation tasks (Glorot et al., 2011; Chen et al., 2012).”
Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning (2017, methodology): “Amazon reviews: For our first experiment we use the Amazon reviews data set with the same preprocessing as used by Chen et al. (2012); Ganin et al. (2016); Louizos et al. (2016).”
A Hybrid Collaborative Filtering Model with Deep Structure for Recommender Systems (2017, background): “Existing literatures have shown that multiple layers stacked together can generate rich representations in hidden layers, and therefore leads to better performance for various tasks (Rifai et al. 2011; Chen et al. 2012; Kavukcuoglu et al. 2009; Glorot, Bordes, and Bengio 2011).”
Projecting Embeddings for Domain Adaption: Joint Modeling of Sentiment Analysis in Diverse Domains (2018, methodology): “For our experiments, we use the 30000 most common uni-and bi-grams as features and take the top 5000 features as pivots (Chen et al., 2012).”
MGAE: Marginalized Graph Autoencoder for Graph Clustering (2017, methodology): “A marginalized denoising autoendcoder (mSDA) was proposed in [4] inspired by SDA.”