Ensembling neural networks: Many could be better than all

By Zhi-Hua Zhou, Jianxin Wu, Wei Tang. 2002

2,013 citations recorded by the source corpus.

Neural network ensemble is a learning paradigm where many neural networks are jointly used to solve a problem. In this paper, the relationship between the ensemble and its component neural networks is analyzed from the context of both regression and classification, which reveals that it may be better to ensemble many instead of all of the neural networks at hand. This result is interesting because at present, most approaches ensemble all the available neural networks for prediction. Then, in order to show that the appropriate neural networks for composing an ensemble can be effectively selected from a set of available neural networks, an approach named GASEN is presented. GASEN trains a number of neural networks at first. Then it assigns random weights to those networks and employs genetic algorithm to evolve the weights so that they can characterize to some extent the fitness of the neural networks in constituting an ensemble. Finally it selects some neural networks based on the evolved weights to make up the ensemble. A large empirical study shows that, compared with some popular ensemble approaches such as Bagging and Boosting, GASEN can generate neural network ensembles with far smaller sizes but stronger generalization ability. Furthermore, in order to understand the working mechanism of GASEN, the bias-variance decomposition of the error is provided in this paper, which shows that the success of GASEN may lie in that it can significantly reduce the bias as well as the variance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

Learning to Transduce with Unbounded Memory (2015, background): “In most cases, this procedure does not yield the actual bestperforming model, and in practice a more sophisticated procedure such as ensembling [20] should produce better results.”
Peer Collaborative Learning for Online Knowledge Distillation (2020, neutral): “Neural Network Ensembling is a simple and effective so-lution for improving the generalisation performance of a model [4, 29, 15].”
Dynamic Weighted Majority: An Ensemble Method for Drifting Concepts (2007, methodology): “There have been numerous studies of ensemble methods (e.g., Hansen and Salamon, 1990; Woods et al., 1997; Quinlan, 1996; Zheng, 1998; Opitz and Maclin, 1999; Bauer and Kohavi, 1999; Maclin and Opitz, 1997; Dietterich, 2000; Zhou et al., 2002; Chawla et al., 2004), and we cannot survey them all.”
Class-Incremental Learning: A Survey (2023, background): “There is also research about CIL before the prosperity of deep learning [11].”
Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research (2015, methodology): “An exemplary approach that uses a more elaborate search method is GASEN (Zhou et al., 2002), which approaches (15) by means of a genetic algorithm.”
Improve Computer-Aided Diagnosis With Machine Learning Techniques Using Undiagnosed Samples (2007, background): “According to Zhou et al. [45], large size of ensemble does not necessarily lead to better performance of an ensemble.”
An ensemble of neural networks for weather forecasting (2004, methodology): “2.3 The generalised ensemble method A generalisation to the BEM method is to find weights for each output that minimises the MSE of the ensemble.”
Online Knowledge Distillation via Collaborative Learning (2020, background): “Majority Voting, Stacked Generalization [27], GASEN [33], and Super Learner [12] explicitly select reliable prediction.”
Deep Co-Training for Semi-Supervised Image Recognition (2018, background): “This is also applicable to deep neural networks [46, 47].”
Extracting symbolic rules from trained neural network ensembles (2003, neutral): “Zhou et al. [49] employs genetic algorithm to select a subset of a population of neural networks.”
Gesture recognition for human-robot collaboration: A review (2017, background): “It is clear that the number of gesture recognition papers has increased rapidly since 1994, indicating the growing interests in this ﬁ eld.”
Ensemble-based classifiers (2010, methodology): “After (Zhou et al. 2002) proved the “many-could-be-better-than-all” theorem, it has become wellknown that it is possible to get a small yet strong ensemble.”
Review of soft sensor methods for regression applications (2016, methodology): “semble of NN models was detailed and discussed in [96], where the authors proposed a method for building an ensemble of NN models based on GA.”
A survey on ensemble learning (2019, methodology): “[39] proposed a neural network based ensemble method, where the weights of networks were updated by genetic algorithm.”