A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networks
By Lior Deutsch, Erik Nijkamp, Yu Yang. 2019
16 citations recorded by the source corpus.
Recent work on mode connectivity in the loss landscape of deep neural networks has demonstrated that the locus of (sub-)optimal weight vectors lies on continuous paths. In this work, we train a neural network that serves as a hypernetwork, mapping a latent vector into high-performance (low-loss) weight vectors, generalizing recent findings of mode connectivity to higher dimensional manifolds. We formulate the training objective as a compromise between accuracy and diversity, where the diversity takes into account trivial symmetry transformations of the target network. We demonstrate how to reduce the number of parameters in the hypernetwork by parameter sharing. Once learned, the hypernetwork allows for a computationally efficient, ancestral sampling of neural network weights, which we recruit to form large ensembles. The improvement in classification accuracy obtained by this ensembling indicates that the generated manifold extends in dimensions other than directions implied by trivial symmetries. For computational efficiency, we distill an ensemble into a single classifier while retaining generalization.
Subfields: Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Generative Models
- Jitendra Malik
- William S. Peebles
- Alexei A. Efros
- Ilija Radosavovic
- Tim Brooks
- Richard Zhang
- Jun-Yan Zhu
- Eli Shechtman
Citations in the corpus
- Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “Hypernetworks have subsequently been extended to support sampling different parameter solutions (Krueger et al., 2017; Deutsch et al., 2019; Ratzlaff & Fuxin, 2019).”
- Principled Weight Initialization for Hypernetworks (2020, background): “In fact, weight initialization for hypernetworks was recognized as an outstanding open problem by prior work (Deutsch et al., 2019) that had questioned the suitability of classical initialization methods for hypernetworks.”