Data Dropout: Optimizing Training Data for Convolutional Neural Networks

By Tianyang Wang, Jun Huan, Bo Li. 2018

65 citations recorded by the source corpus.

Deep learning models learn to fit training data while they are highly expected to generalize well to testing data. Most works aim at finding such models by creatively designing architectures and fine-tuning parameters. To adapt to particular tasks, hand-crafted information such as image prior has also been incorporated into end-to-end learning. However, very little progress has been made on investigating how an individual training sample will influence the generalization ability of a model. In other words, to achieve high generalization accuracy, do we really need all the samples in a training dataset? In this paper, we demonstrate that deep learning models such as convolutional neural networks may not favor all training samples, and generalization accuracy can be further improved by dropping those unfavorable samples. Specifically, the influence of removing a training sample is quantifiable, and we propose a Two-Round Training approach, aiming to achieve higher generalization accuracy. We locate unfavorable samples after the first round of training, and then retrain the model from scratch with the reduced training dataset in the second round. Since our approach is essentially different from fine-tuning or further training, the computational cost should not be a concern. Our extensive experimental results indicate that, with identical settings, the proposed approach can boost performance of the well-known networks on both high-level computer vision problems such as image classification, and low-level vision problems such as image denoising.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Exploring Example Influence in Continual Learning (2022, positive): “propose a similar idea and use the influences to reweight or dropout the training data [31, 13, 36].”
G-DAug: Generative Data Augmentation for Commonsense Reasoning (2020, methodology): “While previous work focuses on removing or examining existing training examples (Koh and Liang, 2017; Wang et al., 2018b), we use influence functions to estimate the effect of including an unseen novel example into the training in order to filter out unhelpful synthetic data points to boost the perf”
Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications (2019, methodology): “Taking a different approach, Wang et al. (2018) apply influence functions (Koh & Liang, 2017) to discard training inputs that do not affect learning.”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “…(Arazo et al., 2019), using them only in a self-supervised training objective (Li et al., 2020a), or even completely ignoring them during training (Wang et al., 2018; Talukdar et al., 2021), upweighting or balancing training on atypical, or avoid memorization of noisy labels (Brown et al., 2020),…”