Selection Via Proxy: Efficient Data Selection For Deep Learning

By Cody A. Coleman, Christopher Yeh, Stephen O. Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, Matei A. Zaharia. 2019

497 citations recorded by the source corpus.

Data selection methods such as active learning and core-set selection are useful tools for machine learning on large datasets, but they can be prohibitively expensive to apply in deep learning. Unlike in other areas of machine learning, the feature representations that these techniques depend on are learned in deep learning rather than given, which takes a substantial amount of training time. In this work, we show that we can significantly improve the computational efficiency of data selection in deep learning by using a much smaller proxy model to perform data selection for tasks that will eventually require a large target model (e.g., selecting data points to label for active learning). In deep learning, we can scale down models by removing hidden layers or reducing their dimension to create proxies that are an order of magnitude faster. Although these small proxy models have significantly higher error, we find that they empirically provide useful rankings for data selection that have a high correlation with those of larger models. We evaluate this "selection via proxy" (SVP) approach on several data selection tasks. For active learning, applying SVP to Sener and Savarese [2018]'s recent method for active learning in deep learning gives a 4x improvement in execution time while yielding the same model accuracy. For core-set selection, we show that a proxy model that trains 10x faster than a target ResNet164 model on CIFAR10 can be used to remove 50% of the training data without compromising the accuracy of the target model, making end-to-end training time improvements via core-set selection possible.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Coresets via Bilevel Optimization for Continual Learning and Streaming (2020, background): “Such selection strategies are used among others by [11] and [1].”
SVP-CF: Selection via Proxy for Collaborative Filtering Data (2021, methodology): “Selection-Via-Proxy (SVP) (Coleman et al., 2020) is a leading coreset mining technique for multi-class classification scenarios.”
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining (2023, methodology): “Coleman et al. [11] select examples based on the uncertainty of a small proxy model for active learning, while DoReMi uses DRO on the excess loss with respect to a reference model, and focuses on data mixture reweighting.”
Queried Unlabeled Data Improves and Robustifies Class-Incremental Learning (2022, methodology): “We adopt two representative activate learning methods to choose the “anchors”: (1) maximum entropy sampling (Lewis & Gale, 1994; Settles, 2009) termed as “Entropy”; and (2) core-set selection via proxy (Coleman et al., 2020) named as “SVP”.”
Deep Learning on a Data Diet: Finding Important Examples Early in Training (2021, methodology): “Recent work on pruning data [1, 2], can be placed in the broader context of identifying coresets that allow training to approximately the same accuracy as would be possible with the original data [3–7].”
GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training (2021, background): “Recently, there have been several efforts to make machine learning models more efficient via data subset selection (Wei et al., 2014a; Kaushal et al., 2019; Coleman et al., 2020; Har-Peled & Mazumdar, 2004; Clarkson, 2010; Mirzasoleiman et al., 2020a; Killamsetty et al., 2021).”
DataComp: In search of the next generation of multimodal datasets (2023, background): “[29] C Coleman, C Yeh, S Mussmann, B Mirzasoleiman, P Bailis, P Liang, J Leskovec, and M Zaharia.”
dcbench: a benchmark for data-centric AI systems (2022, background): “Another name used for this problem is "core-set selection" [4].”