Issues in Stacked Generalization

By Kai Ming Ting, Ian H. Witten. Journal of Artificial Intelligence Research, 2011

791 citations recorded by the source corpus.

Stacked generalization is a general method of using a high-level model to combine lower-level models to achieve greater predictive accuracy. In this paper we address two crucial issues which have been considered to be a 'black art' in classification tasks ever since the introduction of stacked generalization in 1992 by Wolpert: the type of generalizer that is suitable to derive the higher-level model, and the kind of attributes that should be used as its input. We find that best results are obtained when the higher-level model combines the confidence (and not just the predictions) of the lower-level ones. We demonstrate the effectiveness of stacked generalization for combining three different types of learning algorithms for classification tasks. We also compare the performance of stacked generalization with majority vote and published results of arcing and bagging.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5d5a28ea30e03398f7867ac7ef79dd497fb67864

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep forest (2017, methodology): “Based on suggestions from studies about stacking [55,63], we use cross-validation procedure to generate inputs from one grade for the next.”
The Tradeoffs Between Open and Traditional Relation Extraction (2008, background): “Previous studies (Ting and Witten, 1999; Zenko and Dzeroski, 2002; Sigletos et al., 2005) have shown that the probabilities of each class value as estimated by each base-level algorithm are effective features when training meta-learners.”
A Review on Multi-Label Learning Algorithms (2014, background): “The major difference lies in that Stacked Aggregation does exploit correlations between labels while the risk of overfitting could be high due to the nature of stacked generalization [16], [147], [173].”
Privacy-preserving Transfer Learning for Knowledge Sharing (2018, background): “The output { ¯ w s As the learning guarantee of stacked generalization is still an open issue (Ting and Witten 1999), we cannot offer the guarantee of generalization performance for Algorithm 4 here.”
Model averaging in ecology: a review of Bayesian, information-theoretic, and tactical approaches for predictive inference (2018, background): “1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 RMSE rjMCMC median 0.00 0.01 0.00 0.11 0.00 0.00 0.08 0.11 0.00 0.14 0.00 0.09 0.14 0.13 0.10 0.09 1.069 BIC 0.00 0.01 0.00 0.18 0.00 0.03 0.17 0.04 0.00 0.19 0.00 0.04 0.24 0.05 0.05 0.01 1.074 Median† – – – – – – – – – – – – – – – – 1.075 m10‡ 0 0 0 0 0 0 0 0”
Reconciling schemas of disparate data sources: a machine-learning approach (2001, methodology): “Training the meta-learner proceeds as follows [23].”
Learning to match ontologies on the Semantic Web (2003, positive): “However, they can also be set automatically using a machine learning approach called stacking [48,45], as we have shown in [12].”
Ontology Matching: A Machine Learning Approach (2004, methodology): “However, they can also be set automatically using a machine learning approach called stacking [36, 33], as we have shown in [11].”
Advances in Intelligent Data Analysis V (2003, methodology): “XOR), we have extended the learning system DIPOL ([10,12,15]) in order to handle example dependent costs.”
Is Combining Classifiers with Stacking Better than Selecting the Best One? (2004, methodology): “• DT: the decision table majority algorithm of Kohavi (1995), • MLR: the multi-response linear regression algorithm, as used by Ting and Witten (1999) and described in Section 2.”
Ensemble Learning (2019, neutral): “Empirical studies on popular ensemble methods can be found in many papers such as [9, 10, 11].”
ECG beat classifier designed by combined neural network model (2005, background): “The learning system should then seek to transform the representations into a space that is easier for learning purposes [25–27].”
Using Stacking to Average Bayesian Predictive Distributions (with Discussion) (2017, methodology): “, n), and each model Mk has a parametric form ŷk = fk(x|θk), stacking is done in two steps (Ting and Witten, 1999).”
Learning to map between ontologies on the semantic web (2002, methodology): “However, they can also be set automatically using a machine learning approach called stacking [37, 34], as we have shown in [12].”