GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training
By Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, Abir De, Rishabh K. Iyer. 2021
337 citations recorded by the source corpus.
The great success of modern machine learning models on large datasets is contingent on extensive computational resources with high financial and environmental costs. One way to address this is by extracting subsets that generalize on par with the full data. In this work, we propose a general framework, GRAD-MATCH, which finds subsets that closely match the gradient of the training or validation set. We find such subsets effectively using an orthogonal matching pursuit algorithm. We show rigorous theoretical and convergence guarantees of the proposed algorithm and, through our extensive experiments on real-world datasets, show the effectiveness of our proposed framework. We show that GRAD-MATCH significantly and consistently outperforms several recent data-selection algorithms and achieves the best accuracy-efficiency trade-off. GRAD-MATCH is available as a part of the CORDS toolkit: \url{https://github.com/decile-team/cords}.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Percy Liang
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Mohammad Rastegari
- Tengyu Ma
- Graham Neubig
In Computer Vision and Pattern Recognition
- Yi Yang
- Luc Van Gool
- Quoc V. Le
- Dacheng Tao
- Chunhua Shen
- Trevor Darrell
- Xiaogang Wang
- Alan L. Yuille
Citations in the corpus
- DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining (2023, background): “(Kaushal et al., 2019, Killamsetty et al., 2021a,b,c, Mirzasoleiman et al., 2020, Paul et al., 2021, Sener and Savarese, 2018, Sorscher et al., 2022, Wang et al., 2020, Wei et al., 2015).”
- CVNets: High Performance Library for Computer Vision (2022, methodology): “Moreover, these methods are compute- and memory-intensive, and do not scale well to large models and datasets [9].”