Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples

By Haw-Shiuan Chang, Eric Learned-Miller, Andrew McCallum. 2017

389 citations recorded by the source corpus.

Self-paced learning and hard example mining re-weight training instances to improve learning accuracy. This paper presents two improved alternatives based on lightweight estimates of sample uncertainty in stochastic gradient descent (SGD): the variance in predicted probability of the correct class across iterations of mini-batch SGD, and the proximity of the correct class probability to the decision threshold. Extensive experimental results on six datasets show that our methods reliably improve accuracy in various network architectures, including additional gains on top of other popular training techniques, such as residual learning, momentum, ADAM, batch normalization, dropout, and distillation.

Subfields: Artificial Intelligence, Optimization & Training, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Statistical Learning Theory

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Other approaches consider training loss (Han et al., 2018; Arazo et al., 2019; Shen & Sanghavi, 2019), confidence (Hovy et al., 2013), prediction variance (Chang et al., 2017), and area under the curve (Pleiss et al., 2020).”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “to capture a distribution, especially for some hard examples [21], [61], [148].”
On Robustness of Neural Architecture Search Under Label Noise (2020, methodology): “Great progress has been made in research on the robustness of learning algorithms under corrupted labels (Arpit et al., 2017; Chang et al., 2017; Ghosh et al., 2017; Patrini et al., 2017; Zhang et al., 2017, 2018; Jiang et al., 2018; Ren et al., 2018; Zhang and Sabuncu, 2018).”
Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference (2021, methodology): “Inspired by works on sample importance (Kloek & Van Dijk, 1978; LeCun et al., 1990; Chang et al., 2017; Katharopoulos & Fleuret, 2018; Csiba & Richt´arik, 2018), we propose a novel sampling strategy speciﬁc for CL that removes samples from the memory based on sample-wise importance as following…”
An Empirical Study of Example Forgetting during Deep Neural Network Learning (2018, background): “This seems to align with the observations in Chang et al. (2017), where the authors re-weight training examples by accounting for the variance of their predictive distribution.”
Learning to Reweight Examples for Robust Deep Learning (2018, methodology): “…weights are typically calculated based on the training loss, as in many classical algorithms such as AdaBoost (Freund & Schapire, 1997), hard negative mining (Malisiewicz et al., 2011), self-paced learning (Kumar et al., 2010), and other more recent work (Chang et al., 2017; Jiang et al., 2017).”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, background): “Perhaps our measures are the closest to those from Chang et al. (2017); they propose prediction variance and threshold closeness—which correspond to variability and conﬁdence , respectively.”
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels (2017, background): “ self-paced learning which favors training samplesofsmallerloss. Afterthat, manypredenedcurriculums were proposed, e.g. , in (Supancic & Ramanan,2013; Jiang et al.,2014;2015;Sangineto et al.,2016;Chang et al., 2017;Ma et al.,2017a;b). For example, Jiang et al. (2014) introduced a curriculum of u”
Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting (2019, background): “For example, SPL [26] and its extensions [28, 29], iterative reweighting [27, 17] and other recent work [46, 30], pay more focus on easy samples with smaller losses.”
Information-Theoretic Measures of Dataset Difficulty (2021, methodology): “It follows earlier work that used the training loss (Han et al., 2018; Arazo et al., 2019; Shen and Sanghavi, 2019), conﬁdence (Hovy et al., 2013), prediction variance (Chang et al., 2017), and area under the curve (Pleiss et al., 2020) to differentiate instances.”
Training Deep Models Faster with Robust, Approximate Importance Sampling (2018, background): “And yet others prioritize examples closest to the model’s decision boundary [17].”
SELFIE: Refurbishing Unclean Samples for Robust Deep Learning (2019, methodology): “ActiveBias (Chang et al., 2017) corrects the backward loss of training samples by prediction variance.”
Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications (2019, methodology): “Chang et al. (2017) propose to train on points with high prediction variance or whose average prediction is close from the decision threshold.”
A Survey on Curriculum Learning (2021, background): “Besides, Chang et al. [8] propose to select the most uncertain examples according to the prediction history, which is consistent with the variance reduction strategies in active learning [100].”
Robust Learning by Self-Transition for Handling Noisy Labels (2020, background): “Discarding Hard Sample: True-labeled samples with large losses are simply discarded by the small-loss trick, though they have a great impact on generalization [4].”
On The Power of Curriculum Learning in Training Deep Networks (2019, background): “Still, all approaches show benefit under different empirical settings (Chang et al., 2017; Zhang et al., 2017).”
Data collection and quality challenges in deep learning: a data-centric AI perspective (2021, methodology): “ActiveBias performs a forward step on a given mini-batch and computes the sample importance for each sample.”