Adaptation Regularization: A General Framework for Transfer Learning

By Mingsheng Long, Jianmin Wang, Guiguang Ding, Sinno Jialin Pan, Philip S. Yu. IEEE Transactions on Knowledge and Data Engineering, 2014

563 citations recorded by the source corpus.

Domain transfer learning, which learns a target classifier using labeled data from a different distribution, has shown promising value in knowledge discovery yet still been a challenging problem. Most previous works designed adaptive classifiers by exploring two learning strategies independently: distribution adaptation and label propagation. In this paper, we propose a novel transfer learning framework, referred to as Adaptation Regularization based Transfer Learning (ARTL), to model them in a unified way based on the structural risk minimization principle and the regularization theory. Specifically, ARTL learns the adaptive classifier by simultaneously optimizing the structural risk functional, the joint distribution matching between domains, and the manifold consistency underlying marginal distribution. Based on the framework, we propose two novel methods using Regularized Least Squares (RLS) and Support Vector Machines (SVMs), respectively, and use the Representer theorem in reproducing kernel Hilbert space to derive corresponding solutions. Comprehensive experiments verify that ARTL can significantly outperform state-of-the-art learning methods on several public text and image datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/02eed03bdf0c5f9c8ad1f2a9dfc9fc8d8e6c817e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Asymmetric Transfer Network for Unbalanced Domain Adaptation (2018, methodology): “Nowadays, deep learning models have been successfully applied to many applications (Krizhevsky and Hinton 2011; Mikolov et al. 2010; Wang, Cui, and Zhu 2016).”
A Comprehensive Survey on Transfer Learning (2019, background): “[39] proposes a general framework termed adaptation regularization-based transfer learning (ARTL).”
Spectral Bisection Tree Guided Deep Adaptive Exemplar Autoencoder for Unsupervised Domain Adaptation (2016, methodology): “Recently, regularizers such as low-rank constraint (Shao, Kit, and Fu 2014) and Maximum Mean Discrepancy (MMD) (Pan et al. 2011; Baktashmotlagh et al. 2013; Long et al. 2014c) have been applied on transfer learning to guide the subspace or kernel learning.”
Towards Out-Of-Distribution Generalization: A Survey (2021, background): “In addition to domain adversarial learning, numerous studies [248, 146, 169, 168, 167, 269] propose the alignment of features to cultivate domain invariant representations.”
Where and How to Transfer: Knowledge Aggregation-Induced Transferability Perception for Unsupervised Domain Adaptation (2021, methodology): “Consequently, the knowledge aggregation-induced transferability ~ Q 2 R jVj is formulated as where D 2 R jVj(cid:4)jVj is the diagonal degree matrix with entries D ii ¼ P j A ij , and D (cid:6) 12 is used to normalize the adjacency matrix A [2], [31].”
Deep Model Based Domain Adaptation for Fault Diagnosis (2017, methodology): “One is the CA on the test data, which is widely adopted in the published literature [5], [23], [25], [26].”
Online and Offline Domain Adaptation for Reducing BCI Calibration Effort (2017, methodology): “5) ARRLS, which was proposed in [25] (manifold regularization was removed), and is also the wAR algorithm introduced in Algorithm 1, by settingwt = ws,i = wt,i = 1.”
Transfer Learning for EEG-Based Brain–Computer Interfaces: A Review of Progress Made Since 2016 (2020, unknown): “Chen et al. [116] integrated feature selection and an adaptation regularization-based TL (ARTL) [48] classiﬁer for cross-subject driver status classiﬁcation.”
Deep Transfer Network: Unsupervised Domain Adaptation (2015, methodology): “It directly designs an adaptive classiﬁer by incorporating distribution adaptation with model regularization [7, 23, 20].”
A Unified Framework for Metric Transfer Learning (2017, methodology): “using the Euclidean distance may be suboptimal: commonly they would not maximize the inter-class distances while minimizing intra-class distances [18], [19], [20], [21].”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, background): “into the DIP [65] respectively SIE model [66]. This term encourages the source samples from the same class to be clustered in the latent space. The Adaptation Regularization based Transfer Learning11 [71] performs DA by optimizing simultaneously the structural risk functional, the joint distribution”
Joint distribution optimal transportation for domain adaptation (2017, methodology): “We compare our method with four other methods: the surrogate kernel approach ([4], denoted SurK), subspace adaptation for its simplicity and good performances on visual adaptation ([8], SA), Adaptation Regularization based Transfer Learning ([30], ARTL), and the two variants of regularized optimal t”