Principled Weight Initialization for Hypernetworks

By Oscar Chang, Lampros Flokas, Hod Lipson. 2020

90 citations recorded by the source corpus.

Hypernetworks are meta neural networks that generate weights for a main neural network in an end-to-end differentiable manner. Despite extensive applications ranging from multi-task learning to Bayesian deep learning, the problem of optimizing hypernetworks has not been studied to date. We observe that classical weight initialization methods like Glorot & Bengio (2010) and He et al. (2015), when applied directly on a hypernet, fail to produce weights for the mainnet in the correct scale. We develop principled techniques for weight initialization in hypernets, and show that they lead to more stable mainnet weights, lower training loss, and faster convergence.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Posterior Meta-Replay for Continual Learning (2021, positive): “Chang et al. (2020)), that we adapted for chunked hypernetworks.”
Block Contextual MDPs for Continual Learning (2021, methodology): “Speciﬁcally, they proposed an algorithm called HyperDynamics where they use a HyperNetwork (Ha et al., 2016; Chang et al., 2019; Klocek et al., 2019; Meyerson & Miikkulainen, 2019) to generate the weights of the dynamics model.”
Parameter Prediction for Unseen Deep Architectures (2021, background): “Chang et al. [59] showed that when the parameters of the network are instead predicted by a hypernetwork, the activations in the network tend to explode or vanish.”