Popular Ensemble Methods: An Empirical Study

By Richard Maclin, David W. Opitz. 1999

An ensemble consists of a set of individually trained classifiers (such as neural networks or decision trees) whose predictions are combined when classifying novel instances. Previous research has shown that an ensemble is often more accurate than any of the single classifiers in the ensemble. Bagging (Breiman, 1996c) and Boosting (Freund & Schapire, 1996; Schapire, 1990) are two relatively new but popular methods for producing ensembles. In this paper we evaluate these methods on 23 data sets using both neural networks and decision trees as our classification algorithm. Our results clearly indicate a number of conclusions. First, while Bagging is almost always more accurate than a single classifier, it is sometimes much less accurate than Boosting. On the other hand, Boosting can create ensembles that are less accurate than a single classifier - especially when using neural networks. Analysis indicates that the performance of the Boosting methods is dependent on the characteristics of the data set being examined. In fact, further results show that Boosting ensembles may overfit noisy data sets, thus decreasing its performance. Finally, consistent with previous studies, our work suggests that most of the gain in an ensemble's performance comes in the first few classifiers combined; however, relatively large gains can be seen up to 25 classifiers when Boosting decision trees.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “A well-known concept in ML is ensemble learning (Opitz & Maclin, 1999; Polikar, 2006; Rokach, 2010).”
Machine Unlearning (2019, background): “Because this departs from prior work on ML ensembles, which typically involves an ensemble of heterogeneous models [42], we may not obtain as large beneﬁts from agggregation as is typically the case.”
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning (2020, background): “we prove that self-distillation can also be viewed as implicitly combining ensemble and knowledge distillation arXiv:2012.09816v1 [cs.LG] 17 Dec 2020 to improve test accuracy. 1 Introduction Ensemble [26, 42, 51, 67, 70, 70, 71, 73, 90] (aka. model averaging) is one of the oldest and most powerful t”
A constructive algorithm for training cooperative neural network ensembles (2003, methodology): “For each tenfold cross-validation, they first partitioned the original dataset into ten equal-sized sets, then each set was in turn used as the test set while the remaining nine sets were used for training individual NNs in an ensemble [40].”
Machine Learning Algorithms: A Review (2022, background): “It has been observed that, a collection of learners is almost always better at doing a particular job rather than individual learners [20].”
A Survey on Distributed Machine Learning (2019, methodology): “At prediction time, the outputs of those instances can then be combined through standard ensemble model aggregation [112].”
Selecting appropriate machine learning methods for digital soil mapping (2020, result): “stronger predictive performance than results produced from any of the models individually [43].”
Constructing Diverse Classifier Ensembles using Artificial Training Examples (2003, methodology): “Decision-tree induction has been the most commonly used base learner in other ensemble studies, but there has been some work using neural networks and naive Bayes [Bauer and Kohavi, 1999; Opitz and Maclin, 1999].”
Dynamic Weighted Majority: An Ensemble Method for Drifting Concepts (2007, background): “There have been numerous studies of ensemble methods (e.g., Hansen and Salamon, 1990; Woods et al., 1997; Quinlan, 1996; Zheng, 1998; Opitz and Maclin, 1999; Bauer and Kohavi, 1999; Maclin and Opitz, 1997; Dietterich, 2000; Zhou et al., 2002; Chawla et al., 2004), and we cannot survey them all.”
An empirical analysis of dropout in piecewise linear networks (2013, methodology): “Dropout training is similar to bagging (Breiman, 1994) and related ensemble methods (Opitz and Maclin, 1999).”
SHIFTX2: significantly improved protein chemical shift prediction (2011, methodology): “There are two popular ensemble methods: Bagging (Breiman 1996) and Boosting (Schapire 1990) or its variants (e.g. Ada-Boost) (Freund and Schapire 1997; Kotsiantis 2007; Opitz and Maclin 1999).”
Ensemble Classification and Regression-Recent Developments, Applications and Future Directions [Review Article] (2016, background): “For example, in meteorology, ensemble forecasting [33] alters the initial conditions of the differential equations to model the meteorology.”
Lung cancer cell identification based on artificial neural network ensembles (2002, background): “However, some researchers prefer another definition, i.e. an artificial neural network ensemble consists of a set of individually trained artificial neural networks whose predictions are combined when classifying novel instances [21].”
Classification in the Presence of Label Noise: A Survey (2014, background): “Several studies have shown that boosting [98] is affected by label noise [99]–[102].”
Hate speech detection: Challenges and solutions (2019, background): “Combining machine learning classifiers is not a new concept [35].”