The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification

By Cheng Ju, Aurélien F. Bibaut, M. J. van der Laan. Journal of Applied Statistics, 2017

389 citations recorded by the source corpus.

Artificial neural networks have been successfully applied to a variety of machine learning tasks, including image recognition, semantic segmentation, and machine translation. However, few studies fully investigated ensembles of artificial neural networks. In this work, we investigated multiple widely used ensemble methods, including unweighted averaging, majority voting, the Bayes Optimal Classifier, and the (discrete) Super Learner, for image recognition tasks, with deep neural networks as candidate algorithms. We designed several experiments, with the candidate algorithms being the same network structure with different model checkpoints within a single training process, networks with same structure but trained multiple times stochastically, and networks with different structure. In addition, we further studied the overconfidence phenomenon of the neural networks, as well as its impact on the ensemble methods. Across all of our experiments, the Super Learner achieved best performance among all the ensemble methods in this study.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/d2a56c13463b021d5f6af4f9f93aa3a5a31cf82d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bernt Schiele
2. Yaoyao Liu
3. Qianru Sun
4. Ping Luo
5. M. A. Ganaie
6. Mohammad Tanveer
7. Minghui Hu
8. Qiushan Guo

Citations in the corpus

• Online Knowledge Distillation via Collaborative Learning (2020, background): “Majority Voting, Stacked Generalization [27], GASEN [33], and Super Learner [12] explicitly select reliable prediction.”
• Ensemble deep learning in bioinformatics (2020, background): “Indeed, empirical results of ensemble methods that combine the output of multiple deep learning models have been shown to achieve better generalizability than a single model 22 .”
• An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, methodology): “Stacking combines multiple models by learning a combiner and it applies to both tasks in supervised learning [6, 29, 54] and unsupervised learning [63].”
• Ensemble deep learning: A review (2021, methodology): “However, when the ensemble contains heterogeneous base learners naive unweighted averaging may result in suboptimal performance as it is affected by the performance of the weak learners and the overconfident learners [122].”