Diversity creation methods: a survey and categorisation

By Gavin Brown, Jeremy L. Wyatt, Rachel Harris, Xin Yao. Information Fusion, 2004

1,095 citations recorded by the source corpus.

Ensemble approaches to classication and regression have attracted a great deal of interest in recent years. These methods can be shown both theoretically and empirically to outperform single predictors on a wide range of tasks. One of the elements required for accurate prediction when using an ensemble is recognised to be error \diversity". However, the exact meaning of this concept is not clear from the literature, particularly for classication tasks. In this paper we rst review the varied attempts to provide a formal explanation of error diversity, including several heuristic and qualitative explanations in the literature. For completeness of discussion we include not only the classication literature but also some excerpts of the rather more mature regression literature, which we believe can still provide some insights. We proceed to survey the various techniques used for creating diverse ensembles, and categorise them, forming a preliminary taxonomy of diversity creation methods. As part of this taxonomy we introduce the idea of implicit and explicit diversity creation methods, and three dimensions along which these may be applied. Finally we propose some new directions that may prove fruitful in understanding classication error diversity.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/5fe2cba2a84414be2508246c6dd29b561ead10f9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Fusion Techniques for Reliable Information: A Survey (2010, background): “Diversity is not related to the ensemble accuracy unequivocally [18].”
Ensemble learning for data stream analysis: A survey (2017, background): “• Dedicated diversity measures for data stream classifier ensembles: recall that ensemble diversity is one of the important characteristics of ensembles in the standard, static data context [25, 110, 161].”
Random forests: from early developments to recent advancements (2014, background): “This explains why many ensemble methods seek to promote diversity among the models they combine (Adeva, Beresi, & Calvo, 2005; Brown, Wyatt, Harris, & Yao, 2005).”
Ensemble Classification and Regression-Recent Developments, Applications and Future Directions [Review Article] (2016, background): “The proof is based on ambiguity decomposition [11], [12].”
Automatic Target Recognition in Synthetic Aperture Radar Imagery: A State-of-the-Art Review (2016, positive): “On the one hand, this motivated researchers to utilize random algorithms such as random decision trees to produce a powerful ensemble [96].”
A survey of multiple classifier systems as hybrid systems (2014, background): “Pairwise measures averaging a measure between each classifier pair in an ensemble, such as Q-statistic [58], kappa-statistics [63], disagreement [64] and double-fault measure [61,65].”
Hierarchical Ensemble of Global and Local Classifiers for Face Recognition (2009, methodology): “In other words, the saliency map of a certain face image is determined in the testing phase (i.e., in an online manner), whereas the discriminative image patches in our method are determined in the training phase (i.e., in an offline manner).”
From dynamic classifier selection to dynamic ensemble selection (2008, background): “Static selection schemes select an EoC for all test patterns, and dynamic selection schemes select different classifiers for different test patterns.”
Ensemble approaches for regression: A survey (2012, methodology): “For neural networks, the methods based on negative correlation (e.g., EENCL [Liu et al. 2000]) are particularly 10:16 J. Mendes-Moreira et al. appealing, due to their theoretical foundations [Brown et al. 2005b] and successful empirical results.”
The Impact of Diversity on Online Ensemble Learning in the Presence of Concept Drift (2010, background): “Although there are some theoretical results, they are highly restricted and make strong assumptions unlikely to be hold in practice [39].”
Pareto-Based Multiobjective Machine Learning: An Overview and Case Studies (2008, background): “Previously , the diversity of the ensemble members has been promoted through the use of different data, different learning algorithms or different learning models [54].”
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning (2020, background): “When training a linear function over such NTK features, traditional theorems [1, 15, 18, 81] suggest that the ensemble of L independently trained models can indeed improve test-time performance, as it enlarges the feature space from ΦW0(x) to {ΦW (i) 0 (x)}i∈[L] for L many independently”