Dataset Distillation

By Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, Alexei A. Efros. 2018

376 citations recorded by the source corpus.

Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to original performance with only a few gradient descent steps, given a fixed network initialization. We evaluate our method in various initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods.

Subfields: Artificial Intelligence, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

Citations in the corpus

Adversarial Filters of Dataset Biases (2020, methodology): “We evaluate on four such out-of-distribution datasets: HANS (McCoy et al., 2019b), NLI Diagnostics (Wang et al., 2018a), Stress tests (Naik et al., 2018) and Adversarial-NLI (Nie et al., 2019), see Appendix § A.3 for details.”
Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, methodology): “Dataset distillation is related to this idea, and uses a bi-level optimization scheme to learn a small subset of pseudo images, such that training on those images yields representations that generalize well to real data (Wang et al., 2018).”
Knowledge Distillation: A Survey (2020, methodology): “…extended to other visual recognition applications, including face recognition (Luo et al. 2016; Kong et al. 2019; Yan et al. 2019; Ge et al. 2018; Wang et al. 2018b, 2019c; Duong et al. 2019; Wu et al. 2020; Wang et al. 2017; Zhang et al. 2020b; Wang et al. 2020b), image/video segmentation (He…”
Mnemonics Training: Multi-Class Incremental Learning Without Forgetting (2020, background): “xemplars are optimizable, and can be regarded as synthesized, while our approach is based on the direct parameterization of exemplars without training extra models. Bilevel optimization program (BOP) [6,34,35] aims to solve two levels of problems in one framework where the Alevel problem is the cons”
Using Hindsight to Anchor Past Knowledge in Continual Learning (2019, methodology): “Finally, our gradient-based learning of anchors bears a similarity to (Simonyan, Vedaldi, and Zisserman 2014) and (Wang et al. 2018).”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “However a few representationsmentioned are transductive in the sense that thev literally corresponds to data points [152], labels [156], or per-”
Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning (2019, methodology): “Some of the previous GFSL approaches (Hariharan and Girshick 2017; Wang et al. 2018; Gao et al. 2018) apply the exemplar-based classification paradigms on both seen and unseen categories to resolve the transductive learning problem, which requires recomputing the centroids for seen categories after…”
Concealed Data Poisoning Attacks on NLP Models (2021, methodology): “, 2017) and distillation (Wang et al., 2018), and approximate the inner training loop using a small number of gradient descent steps.”
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (2023, background): “Question bootstrapping also has an intrinsic connection to dataset distillation [65, 72] and machine teaching [35, 36, 52, 75], where the shared target is to construct a training dataset that best facilitates generalization.”
Few-shot learning with adaptively initialized task optimizer: a practical meta-learning approach (2019, background): “Forexample,imagesofrarespecies arehardtocollect,thusthemodelshouldbeabletodovisualrecognitionbasedonsingleor a few reference examples (Wang et al. 2018b); it is cumbersome to require a user recording multiple facial expressions into a system in advance (Tan et al. 2006); for a robot, imitating from…”
Customizing Triggers with Concealed Data Poisoning (2020, positive): “, 2017) and distillation (Wang et al., 2018), and approximate the inner training loop using a small number of gradient descent steps.”
Soft-Label Dataset Distillation and Text Dataset Distillation (2019, methodology): “Additionally, [4] suggest that the above methods can work with multiple gradient descent (GD) steps and with multiple epochs over the distilled data.”