Dataset Distillation with Infinitely Wide Convolutional Networks

By Timothy Nguyen, Roman Novak, Lechao Xiao, Jaehoon Lee. 2021

307 citations recorded by the source corpus.

The effectiveness of machine learning algorithms arises from being able to extract useful features from large amounts of data. As model and dataset sizes increase, dataset distillation methods that compress large datasets into significantly smaller yet highly performant ones will become valuable in terms of training efficiency and useful feature extraction. To that end, we apply a novel distributed kernel based meta-learning framework to achieve state-of-the-art results for dataset distillation using infinitely wide convolutional neural networks. For instance, using only 10 datapoints (0.02% of original dataset), we obtain over 65% test accuracy on CIFAR-10 image classification task, a dramatic improvement over the previous best test accuracy of 40%. Our state-of-the-art results extend across many other settings for MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and SVHN. Furthermore, we perform some preliminary analyses of our distilled datasets to shed light on how they differ from naturally occurring data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Information-theoretic Online Memory Selection for Continual Learning (2022, neutral): “Dataset distillation (Zhao et al., 2020; Nguyen et al., 2021) aims at a small set training on which achieves similar performances to training on the original dataset.”
Dataset Distillation by Matching Training Trajectories (2022, methodology): “Subsequently, several works signiﬁcantly improved the results by learning soft labels [2, 38], amplifying learning signal via gradient matching [47], adopting augmentations [45], and optimizing with respect to the inﬁnite-width kernel limit [25, 26].”
Dataset Condensation with Distribution Matching (2021, methodology): “More detailed comparison and discussion to other methods [39, 31, 32], MMD baseline [18] and generative baselines including DC-VAE [33], BigGAN [8] and GMMN [28] can be found in the supplementary.”