Dataset Condensation with Gradient Matching

By Bo Zhao, Konda Reddy Mopuri, Hakan Bilen. 2020

720 citations recorded by the source corpus.

Efficient training of deep neural networks is an increasingly important problem in the era of sophisticated architectures and large-scale datasets. This paper proposes a training set synthesis technique, called Dataset Condensation, that learns to produce a small set of informative samples for training deep neural networks from scratch in a small fraction of the required computational cost on the original data while achieving comparable results. We rigorously evaluate its performance in several computer vision benchmarks and show that it significantly outperforms the state-of-the-art methods. Finally we show promising applications of our method in continual learning and domain adaptation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Information-theoretic Online Memory Selection for Continual Learning (2022, neutral): “Dataset distillation (Zhao et al., 2020; Nguyen et al., 2021) aims at a small set training on which achieves similar performances to training on the original dataset.”
Meta-Learning in Neural Networks: A Survey (2020, background): “In dataset distillation [152], [153], [154], the support images themselves are learned such that a few steps on”
Dataset Condensation with Distribution Matching (2021, methodology): “However, to this end, we do not limit our method to select a subset of the training samples but to synthesize them as in [45, 50].”
Dataset Distillation by Matching Training Trajectories (2022, methodology): “To reduce the optimization difﬁculty, other methods [45, 47] focus on short-range behavior, enforcing a single training step on distilled data to match that on real data.”
Rethinking Search: Making Experts out of Dilettantes (2021, background): ““condense” or “distill” massive training datasets into smaller, more manageable ones, with little loss in information will likely need to be employed [96, 106].”
Dataset Distillation with Infinitely Wide Convolutional Networks (2021, methodology): “Overall, our transfer to finite-width networks outperforms prior art based on DC/DSA [Zhao et al., 2021, Zhao and Bilen, 2021] in the 1 image per class setting for all the RGB datasets (SVHN, CIFAR-10, CIFAR-100).”
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (2023, background): “Question bootstrapping also has an intrinsic connection to dataset distillation [73, 82] and machine teaching [40, 41, 58, 85], where the shared target is to construct a training dataset that best facilitates generalization.”
Dataset Condensation with Differentiable Siamese Augmentation (2021, methodology): “…a recent research direction, training set synthesis aims at generating a small training set which is further used to train deep neural networks for the downstream task (Wang et al., 2018; Sucholutsky & Schonlau, 2019; Bohdal et al., 2020; Such et al., 2020; Nguyen et al., 2021; Zhao et al., 2021).”