GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning

By Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, R. Dallas, Indian Institute of Technology Bombay Institution One, I. Two. 2020

324 citations recorded by the source corpus.

Large scale machine learning and deep models are extremely data-hungry. Unfortunately, obtaining large amounts of labeled data is expensive, and training state-of-the-art models (with hyperparameter tuning) requires significant computing resources and time. Secondly, real-world data is noisy and imbalanced. As a result, several recent papers try to make the training process more efficient and robust. However, most existing work either focuses on robustness or efficiency, but not both. In this work, we introduce GLISTER, a GeneraLIzation based data Subset selecTion for Efficient and Robust learning framework. We formulate GLISTER as a mixed discrete-continuous bi-level optimization problem to select a subset of the training data, which maximizes the log-likelihood on a held-out validation set. We then analyze GLISTER for simple classifiers such as gaussian and multinomial naive-bayes, k-nearest neighbor classifier, and linear regression and show connections to submodularity. Next, we propose an iterative online algorithm GLISTER-ONLINE, which performs data selection iteratively along with the parameter updates, and can be applied to any loss-based learning algorithm. We then show that for a rich class of loss functions including cross-entropy, hinge-loss, squared-loss, and logistic-loss, the inner discrete data selection is an instance of (weakly) submodular optimization, and we analyze conditions for which GLISTER-ONLINE reduces the validation loss and converges. Finally, we propose GLISTER-ACTIVE, an extension to batch active learning, and we empirically demonstrate the performance of GLISTER on a wide range of tasks including, (a) data selection to reduce training time, (b) robust learning under label noise and imbalance settings, and (c) batch-active learning with a number of deep and shallow models. We show that our framework improves upon the state of the art both in efficiency and accuracy (in cases (a) and (c)) and is more efficient compared to other state-of-the-art robust learning algorithms in case (b). The code for GLISTER is at: https://github.com/dssresearch/GLISTER.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

GCR: Gradient Coreset based Replay Buffer Selection for Continual Learning (2021, neutral): “Several recent works [35, 36, 37, 38, 39] also use coresets to approximate the gradient sum of the individual samples across the entire dataset, for efﬁcient and robust supervised learning.”
CVNets: High Performance Library for Computer Vision (2022, unknown): “2 Sample efficient training Previous works [8, 9, 11] remove and re-weight data samples to reduce optimization updates (or number of forward passes) at the”
RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning (2021, background): “A single epoch here meant a pass over random mini-batches of data points, such that the total number of data points encountered is equal to the size of the coreset of the unlabeled data.”
GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training (2021, methodology): “…results (c.f., Section 5), we show that GRADMATCH achieves much a better accuracy and training-time trade-off than several state-of-the-art approaches such as CRAIG (Mirzasoleiman et al., 2020a), GLISTER (Killamsetty et al., 2021), random subsets, and even full training with early stopping.”