Dataset Meta-Learning from Kernel Ridge-Regression

By Timothy Nguyen, Zhourung Chen, Jaehoon Lee. International Conference on Learning Representations, 2020

326 citations recorded by the source corpus.

One of the most fundamental aspects of any machine learning algorithm is the training data used by the algorithm. We introduce the novel concept of $\epsilon$-approximation of datasets, obtaining datasets which are much smaller than or are significant corruptions of the original training data while maintaining similar model performance. We introduce a meta-learning algorithm called Kernel Inducing Points (KIP) for obtaining such remarkable datasets, inspired by the recent developments in the correspondence between infinitely-wide neural networks and kernel ridge-regression (KRR). For KRR tasks, we demonstrate that KIP can compress datasets by one or two orders of magnitude, significantly improving previous dataset distillation and subset selection methods while obtaining state of the art results for MNIST and CIFAR-10 classification. Furthermore, our KIP-learned datasets are transferable to the training of finite-width neural networks even beyond the lazy-training regime, which leads to state of the art results for neural network dataset distillation with potential applications to privacy-preservation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8212605d274d5e68bcedf990728f4f5c26f88168

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Dataset Distillation by Matching Training Trajectories (2022, methodology): “Subsequently, several works signiﬁcantly improved the results by learning soft labels [2, 38], amplifying learning signal via gradient matching [47], adopting augmentations [45], and optimizing with respect to the inﬁnite-width kernel limit [25, 26].”
Dataset Condensation with Distribution Matching (2021, methodology): “learning rate for the synthetic images, for learning different sizes of synthetic sets, while existing methods [45, 50, 31, 44, 11] have to tune more hyperparameters such as the steps to update synthetic images and network parameters respectively.”
Dataset Condensation with Differentiable Siamese Augmentation (2021, background): “Note that we are aware of two recent work, Generative Teaching Networks (GTN) (Such et al., 2020) and Kernel Inducing Point (KIP) (Nguyen et al., 2021).”
Dataset Distillation with Infinitely Wide Convolutional Networks (2021, methodology): “By contrast, the approach of dataset distillation is to synthesize datasets that are more informative than their natural counterparts when equalizing for dataset size [Wang et al., 2018, Bohdal et al., 2020, Nguyen et al., 2021, Zhao and Bilen, 2021].”