Knowledge transfer via multiple model local structure mapping

By Jing Gao, Wei Fan, Jing Jiang, Jiawei Han. 2008

349 citations recorded by the source corpus.

The effectiveness of knowledge transfer using classification algorithms depends on the difference between the distribution that generates the training examples and the one from which test examples are to be drawn. The task can be especially difficult when the training examples are from one or several domains different from the test domain. In this paper, we propose a locally weighted ensemble framework to combine multiple models for transfer learning, where the weights are dynamically assigned according to a model's predictive power on each test example. It can integrate the advantages of various learning algorithms and the labeled information from multiple training domains into one unified classification model, which can then be applied on a different domain. Importantly, different from many previously proposed methods, none of the base learning method is required to be specifically designed for transfer learning. We show the optimality of a locally weighted ensemble framework as a general approach to combine multiple models for domain transfer. We then propose an implementation of the local weight assignments by mapping the structures of a model onto the structures of the test domain, and then weighting each model locally according to its consistency with the neighborhood structure around the test example. Experimental results on text classification, spam filtering and intrusion detection data sets demonstrate significant improvements in classification accuracy gained by the framework. On a transfer learning task of newsgroup message categorization, the proposed locally weighted ensemble framework achieves 97% accuracy when the best single model predicts correctly only on 73% of the test examples. In summary, the improvement in accuracy is over 10% and up to 30% across different problems.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Multi-Source Domain Adaptation: A Causal View (2015, methodology): “We include the following representative hypothesis combination methods for comparison: LWE (Gao et al. 2008), convex hypothesis combination (Mansour, Mohri, and Rostamizadeh 2008), denoted convex , simple adapt (Mansour, Mohri, and Rostamizadeh 2008), and dstr wgh (H) , which adopts the distribution”
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (2018, background): “Hierarchical Bayesian models have a long history of use in both transfer learning and domain adaptation (e.g., Lawrence & Platt, 2004; Yu et al., 2005; Gao et al., 2008; Daumé III, 2009; Wan et al., 2012).”
Pre-Trained Models: Past, Present and Future (2021, methodology): “Therefore, these methods pre-encode knowledge into shared model parameters (Lawrence and Platt, 2004; Evgeniou and Pontil, 2004; Williams et al., 2007; Gao et al., 2008), and then transfer the knowledge by fine-tuning pre-trained parameters with the data of”
A Comprehensive Survey on Transfer Learning (2019, background): “[132] proposes another ensemble-based framework that is referred to as locally weighted ensemble (LWE).”
Causal Transfer in Machine Learning (2015, methodology): “This idea originates in the hierarchical Bayesian literature [Bonilla et al., 2007, Gao et al., 2008].”
A Comprehensive Survey of Privacy-preserving Federated Learning (2021, background): “• Parameter-based FTL aims to exploit shared parameters or prior distributions of hyperparameters between the source domain and target domain models to effectively encode the transformation knowledge [60, 74, 136].”
A Survey on Transfer Learning (2010, methodology): “Gao et al. [49] proposed a locally weighted ensemble learning framework to combine multiple models for transfer learning, where the weights are dynamically assigned according to a model's predictive power on each test example in the target domain.”
It's who you know: graph mining using recursive structural features (2011, background): “Representative work includes multi-label text classification [30, 13], crossdomain sentiment prediction [4, 6, 15], intrusion detection [12], verb argument classification [19], cross-lingual classification [25], and cross-domain relation extraction [29].”
Transfer learning using computational intelligence: A survey (2015, background): “4) Cluster-based learning methods rely on the assumption that samples connected by high-density paths are likely to have the same label if there is a high density path between them [43].”
Cross-Domain Learning from Multiple Sources: A Consensus Regularization Perspective (2010, methodology): “Finally, Gao et al. [20] proposed a multiple model local structure mapping scheme that attaches different weights to the models by the local manifold structure of testing samples.”
Multi-source domain adaptation and its application to early detection of fatigue (2011, methodology): “The proposed framework differs from LWE [12] in that in LWE, the label y of an unlabeled target domain data x is computed using a local weighting ensemble (LWE) scheme:”
Cross Validation Framework to Choose amongst Models and Datasets for Transfer Learning (2010, methodology): “5, K-NN and NNge(Ng), and three state-of-theart transfer learning methods: TrAdaBoost(TA) [12], LatentMap(LM) [13] and LWE [14].”
Cross-view action recognition via view knowledge transfer (2011, methodology): “Thus, we employ the Locally Weighted Ensemble (LWE) scheme [18] to dynamically estimate combination weights for every test example by checking the consistency of data structures between the model predictions (near the test example) and the real data distribution (near the test example).”
Invariant Models for Causal Transfer Learning (2015, background): “This idea originates in the hierarchical Bayesian literature (Bonilla et al., 2007; Gao et al., 2008).”
Multisource Transfer Learning for Cross-Subject EEG Emotion Recognition (2020, background): “3) Parameter transfer [27], [28], which assumes the target shares parameters with the source in different tasks.”