Diversity in search strategies for ensemble feature selection

By Alexey Tsymbal, Mykola Pechenizkiy, Padraig Cunningham. Information Fusion, 2005

343 citations recorded by the source corpus.

Ensembles of learnt models constitute one of the main current directions in machine learning and data mining. Ensembles allow us to achieve higher accuracy, which is often not achievable with single models. It was shown theoretically and experimentally that in order for an ensemble to be effective, it should consist of base classifiers that have diversity in their predictions. One technique, which proved to be effective for constructing an ensemble of diverse base classifiers, is the use of different feature subsets, or so-called ensemble feature selection. Many ensemble feature selection strategies incorporate diversity as an objective in the search for the best collection of feature subsets. A number of ways are known to quantify diversity in ensembles of classifiers, and little research has been done about their appropriateness to ensemble feature selection. In this paper, we compare five measures of diversity with regard to their possible use in ensemble feature selection. We conduct experiments on 21 data sets from the UCI machine learning repository, comparing the ensemble accuracy and other characteristics for the ensembles built with ensemble feature selection based on the considered measures of diversity. We consider four search strategies for ensemble feature selection together with the simple random subspacing: genetic search, hill-climbing, and ensemble forward and backward sequential selection. In the experiments, we show that, in some cases, the ensemble feature selection process can be sensitive to the choice of the diversity measure, and that the question of the superiority of a particular measure depends on the context of the use of diversity and on the data being processed. In many cases and on average, the plain disagreement measure is the best. Genetic search, kappa, and dynamic voting with selection form the best combination of a search strategy, diversity measure and integration method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c9f87ad639fcfe4d7e0969830a65818e05eaa8f0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zeyuan Allen-Zhu
2. Yuanzhi Li
3. Jun Zhu
4. Dominik Lzak
5. Georg Peters
6. Pawan Lingras
7. Tianyu Pang
8. Chao Du

In Computer Vision and Pattern Recognition

1. Zeyuan Allen-Zhu
2. Yuanzhi Li
3. Tianyu Pang
4. Jun Zhu
5. Kun Xu
6. Dominik Lzak
7. M. de Vos
8. Y. H. Yang

Citations in the corpus

• Ensembles for feature selection: A review and future trends (2019, neutral): “Although there are several works regarding diversity in ensembles for classi(cid:12)cation [50, 48, 51], there is a necessity for the establishment of novel diversity measures for ensembles for other machine learning algorithms, as feature selection [8].”
• A hybrid genetic algorithm for feature selection wrapper based on mutual information (2007, methodology): “It was first introduced by Cohen (1960) and has been used in ensemble feature selection methods (Opitz, 1999; Tsymbal et al., 2005).”
• XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging (2020, background): “Theoretically, for a joint model to be effective, each individual model should be diversified [31].”
• Rough Sets: Selected Methods and Applications in Management and Engineering (2012, background): “[57], Golan and Edwards [58, 59] applied the rough sets model to discover strong trading rules, which reflect highly repetitive patterns in data, from historical database of the Toronto stock exchange.”
• Survey on speech emotion recognition: Features, classification schemes, and databases (2011, methodology): “In fact, MCS is now a well-established area in pattern recognition [66,67,127,126] and there are many aggregation techniques that have not been applied to speech emotion recognition such as Adaboost.”
• An Analysis of Ensemble Pruning Techniques Based on Ordered Aggregation (2009, positive): “Successful ensemble creation methods need to take into account both accuracy and diversity [28], [42].”
• Improving Adversarial Robustness via Promoting Ensemble Diversity (2019, background): “(1) is also effective to improve ensemble performance in the normal setting (Krogh & Vedelsby, 1995; Tsymbal et al., 2005).”
• A Review of Ensemble Methods in Bioinformatics (2010, background): “From the above analysis, it is clear that in order to obtain an improvement the base classifiers need to be accurate (better than chance) and diverse from each other [17].”
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning (2020, background): “Ensemble works for random features / NTK. Traditional theorems (Alhamdoosh & Wang, 2014; Brown et al., 2005a; Bryll et al., 2003; Tsymbal et al., 2005) suggest that the ensemble of independently trained random feature models can indeed significantly improve test-time performance, as it enlarges the…”