Coresets for Data-efficient Training of Machine Learning Models

By Baharan Mirzasoleiman, J. Bilmes, Jure Leskovec. 2019

541 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Mohit Bansal
2. Quoc V. Le
3. Yoshua Bengio
4. Zhe Gan
5. Yi Yang
6. Mitchell Wortsman
7. Percy Liang
8. Ali Farhadi

Citations in the corpus

• GCR: Gradient Coreset based Replay Buffer Selection for Continual Learning (2021, neutral): “Several recent works [35, 36, 37, 38, 39] also use coresets to approximate the gradient sum of the individual samples across the entire dataset, for efficient and robust supervised learning.”
• Beyond neural scaling laws: beating power law scaling via data pruning (2022, background): “[19]) have been proposed for this, but their computation is expensive, and so data-pruning has been less explored at scale on ImageNet.”
• DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining (2023, background): “When domainsare very fine-grained, itwill be important to control the pessimism of DRO (e.g., DRO can put all the weight on a small set of worst-case examples).”
• DataComp: In search of the next generation of multimodal datasets (2023, background): “[91] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec.”
• GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training (2021, methodology): “This result is new and different from the one shown in (Mirzasoleiman et al., 2020a) since it is for full GD and SGD, and not for incremental gradient descent algorithms discussed in (Mirzasoleiman et al., 2020a).”
• CVNets: High Performance Library for Computer Vision (2022, background): “Previous works [11, 12, 14] remove and re-weight data samples to reduce optimization updates (or number of forward passes) at the cost of performance degradation.”
• Learning to Learn and Remember Super Long Multi-Domain Task Sequence (2022, methodology): “Future work includes online selecting the most informative coreset [49] from the online task stream.”
• RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning (2021, methodology): “The baselines considered are RANDOM (where we just randomly select a subset of unlabeled data points of the same size as RETRIEVE), CRAIG [39, 23] and FULL-EARLYSTOP.”
• GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning (2020, methodology): “For example, the data selection approaches such as (Wei, Iyer, and Bilmes 2015; Mirzasoleiman, Bilmes, and Leskovec 2020; Shinohara 2014) and others focus on approximating either gradients or performance on the training sets, and hence would not be suitable for scenarios such as label noise and…”