HyperGAN: A Generative Model for Diverse, Performant Neural Networks

By Neale Ratzlaff, Fuxin Li. 2019

74 citations recorded by the source corpus.

Standard neural networks are often overconfident when presented with data outside the training distribution. We introduce HyperGAN, a new generative model for learning a distribution of neural network parameters. HyperGAN does not require restrictive assumptions on priors, and networks sampled from it can be used to quickly create very large and diverse ensembles. HyperGAN employs a novel mixer to project prior samples to a latent space with correlated dimensions, and samples from the latent space are then used to generate weights for each layer of a deep neural network. We show that HyperGAN can learn to generate parameters which label the MNIST and CIFAR-10 datasets with competitive performance to fully supervised learning, while learning a rich distribution of effective parameters. We also show that HyperGAN can also provide better uncertainty estimates than standard ensembles by evaluating on out of distribution data as well as adversarial examples.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Neural Architecture Search & AutoML

Citations in the corpus

Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “Hypernetworks have subsequently been extended to support sampling different parameter solutions (Krueger et al., 2017; Deutsch et al., 2019; Ratzlaff & Fuxin, 2019).”
Principled Weight Initialization for Hypernetworks (2020, unknown): “…et al., 2019; Serr ` a et al., 2019; Meyerson & Miikkulainen, 2019), continual learning (von Oswald et al., 2019), generative models (Suarez, 2017; Ratzlaff & Fuxin, 2019), ensemble learning (Kristiadi & Fischer, 2019), hyperpa-rameter optimization (Lorraine & Duvenaud, 2018), and adversarial…”
Parameter Prediction for Unseen Deep Architectures (2021, methodology): “Besides the works discussed in § 6, our work is also loosely related to other parameter prediction methods [113, 98, 114], analysis of graph structure of neural networks [46], knowledge distillation from multiple teachers [115], compression methods [116] and optimization-based initialization [117– 1”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “Data science: GANs have been used in data generating [214], [418]–[426], neural networks generating [427], data augmentation [428], [429], spatial representation learning [430], network embedding [431], heterogeneous information networks [432], and mobile user profiling [433].”