Flexible Dataset Distillation: Learn Labels Instead of Images

By Ondrej Bohdal, Yongxin Yang, Timothy M. Hospedales. 2020

124 citations recorded by the source corpus.

We study the problem of dataset distillation - creating a small set of synthetic examples capable of training a good model. In particular, we study the problem of label distillation - creating synthetic labels for a small set of real images, and show it to be more effective than the prior image-based approach to dataset distillation. Interestingly, label distillation can be applied across datasets, for example enabling learning Japanese character recognition by training only on synthetically labeled English letters. Methodologically, we introduce a more robust and flexible meta-learning algorithm for distillation, as well as an effective first-order strategy based on convex optimization layers. Distilling labels with our new algorithm leads to improved results over prior image-based distillation. More importantly, it leads to clear improvements in flexibility of the distilled dataset in terms of compatibility with off-the-shelf optimizers and diverse neural architectures.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “…for model compression, the idea of knowledge transfer has been further applied in compressing the training data, i.e., dataset distillation, which transfers the knowledge from a large dataset into a small dataset to reduce the training loads of deep models (Wang et al. 2018c; Bohdal et al. 2020).”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “can be used in distilling core sets [155] as in dataset distillation; or semi-supervised learning, for example to directly learn the unlabeled set’s labels to optimize validation set performance [156], [157].”
Dataset Condensation with Differentiable Siamese Augmentation (2021, methodology): “Training set synthesis competitors are Dataset Distillation (DD) (Wang et al., 2018), Label Distillation (LD) (Bohdal et al., 2020), Dataset Condensation (DC) (Zhao et al., 2021) which we built our method on.”
Dataset Distillation with Infinitely Wide Convolutional Networks (2021, methodology): “3 LD [Bohdal et al., 2020] is another baseline which distills only labels using the AlexNet architecture.”
Dataset Condensation with Distribution Matching (2021, methodology): “We also compare to four state-of-the-art training set synthesis methods, namely, DD [45], LD [6], DC [50] and DSA [49].”
Dataset Meta-Learning from Kernel Ridge-Regression (2020, background): “The works Sucholutsky & Schonlau (2019); Bohdal et al. (2020) build upon Wang et al. (2018) by distilling labels.”
Dataset Distillation by Matching Training Trajectories (2022, methodology): “Dataset distillation has become a lively research topic in machine learning [2, 25, 26, 38, 45, 46, 47] with various applications, such as continual learning, neural architecture search, and privacy-preserving ML.”
Dataset Condensation with Gradient Matching (2020, background): “, 2018) and extended in (Sucholutsky & Schonlau, 2019; Bohdal et al., 2020; Such et al., 2020), is to pose the parameters θ as a function of the synthetic data S: S = argmin S L (θ(S)) subject to θ(S) = argmin θ L(θ).”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, background): “The knowledge from a large dataset is transferred to a smaller dataset that signiﬁcantly reduces computational loads of training DL models [193], [194].”