The Power of Ensembles for Active Learning in Image Classification

By William H. Beluch, Tim Genewein, Andreas Nürnberger, Jan M. Köhler. 2018

763 citations recorded by the source corpus.

Deep learning methods have become the de-facto standard for challenging image processing tasks such as image classification. One major hurdle of deep learning approaches is that large sets of labeled data are necessary, which can be prohibitively costly to obtain, particularly in medical image diagnosis applications. Active learning techniques can alleviate this labeling effort. In this paper we investigate some recently proposed methods for active learning with high-dimensional data and convolutional neural network classifiers. We compare ensemble-based methods against Monte-Carlo Dropout and geometric approaches. We find that ensembles perform better and lead to more calibrated predictive uncertainties, which are the basis for many active learning algorithms. To investigate why Monte-Carlo Dropout uncertainties perform worse, we explore potential differences in isolation in a series of experiments. We show results for MNIST and CIFAR-10, on which we achieve a test set accuracy of 90% with roughly 12,200 labeled images, and initial results on ImageNet. Additionally, we show results on a large, highly class-imbalanced diabetic retinopathy dataset. We observe that the ensemble-based active learning effectively counteracts this imbalance during acquisition.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, neutral): “Ensembles of neural networks, even of size three and five, have been shown in some cases to give more reliable uncertainty estimates than other leading approaches such as BNNs (Lakshminarayanan, Pritzel, and Blundell 2017; Beluch et al. 2018; Choi et al. 2016; Snoek et al. 2019; Zaidi et al. 2020).”
Few-Shot Continual Active Learning by a Robot (2022, background): “For object learning, many researchers have presented active learning techniques using uncertainty sampling [35, 18, 14, 15, 36, 16].”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “…heuristics (Roy and McCallum, 2001; Settles and Craven, 2008; Li and Guo, 2013), the latter are nonetheless once again dominant in the era of deep learning (Beluch et al., 2018; Geifman and El-Yaniv, 2019; Gal and Ghahramani, 2015; Srivastava et al., 2014) and the challenges seem to be faced anew.”
Unsupervised label noise modeling and loss correction (2019, background): “Convolutional Neural Networks (CNNs) have recently become the par excellence base approach to deal with many computer vision tasks (DeTone et al., 2016; Ono et al., 2018; Beluch et al., 2018; Redmon et al., 2016; Zhao et al., 2017; Krishna et al., 2017).”
Towards Robust and Reproducible Active Learning using Neural Networks (2020, methodology): “model trained for image classification) within their sampling function [7,25], where as others use different models for task and sampling functions [2,27].”
Learning Novel Objects Continually Through Curiosity (2021, background): “In recent years, attempts have been made to develop active learning techniques to reduce the cost of labelling [7], [8], [9], [10], [11].”
Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds (2019, methodology): “An ensemble of classifiers could also be used to effectively estimate uncertainty (Beluch et al., 2018).”
Learning Loss for Active Learning (2019, methodology): “The performance gaps between these methods are similar to those of [4].”
A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving (2020, methodology): “In general, MC-Dropout provides a practical way to perform approximate inference in Bayesian Neural Netowrks (BNNs), and is scalable to large datasets and network architectures such as in image classification [65] or semantic segmentation [61], [74].”
Ensemble deep learning: A review (2021, background): “[156] discussed the power of ensembles for active learning is significantly better than Monte-Carlo Dropout and geometric approaches.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “• The results of our research show that although a variety of ensemble methods [474], [475], [476], [477], [478], [479], [480] also have a high ability to deal with uncertainty along with good performance and optimizing the performance of other methods, these high capabilities of these methods have ”
Online Learning of Objects through Curiosity-Driven Active Learning (2021, background): “In recent years, attempts have been made to develop active learning techniques to reduce the cost of labelling [7], [8], [9], [10], [11].”
A Survey of Deep Active Learning (2020, background): “small sub-sample set, because the relative difference of sample importance in some datasets is almost negligible; therefore, it is not advisable to blindly use smaller sub-datasets in the AL context. [14] found that compared with the Bayesian deep learning approach (Monte-Carlo dropout [47]) and den”
TRADI: Tracking deep neural network weight distributions (2019, neutral): “active learning [3].”
Knockoff Nets: Stealing Functionality of Black-Box Models (2018, background): “, clustering or estimating graph-density [2,8].”
BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations (2022, methodology): “In order to alleviate this issue, we follow [78] to train an ensemble model of 16 segmentation heads and use the Jensen-Shannon divergence as uncertainty measure to filter out the top 10% most uncertain images [3,44,54].”
Evaluation of Lifelong Learning Systems (2020, background): “…to measure the quality of this adaptation process by considering the dependence between cost of interactions and performance of the system (Krogh and Vedelsby, 1995; Siddhant and Lipton, 2018; Drugman et al., 2019; Beluch et al., 2018; P´erez-Dattari et al., 2018; Celemin and Ruiz-del Solar, 2019).”
Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling (2020, methodology): “Similar to existing work on high dimensional uncertainty estimation (Beluch et al., 2018; Chitta et al., 2018), we use an ensemble {h̃ω(i)}si=1 of s neural networks h̃ω with parameters ω to predict uj given input λj .”