HyperNetworks

By David R Ha, Andrew M. Dai, Quoc V. Le. 2016

1,965 citations recorded by the source corpus.

This work explores hypernetworks: an approach of using one network, also known as a hypernetwork, to generate the weights for another network. We apply hypernetworks to generate adaptive weights for recurrent networks. In this case, hypernetworks can be viewed as a relaxed form of weight-sharing across layers. In our implementation, hypernetworks are are trained jointly with the main network in an end-to-end fashion. Our main result is that hypernetworks can generate non-shared weights for LSTM and achieve state-of-the-art results on a variety of sequence modelling tasks including character-level language modelling, handwriting generation and neural machine translation, challenging the weight-sharing paradigm for recurrent networks.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Neural Machine Translation in Linear Time (2016, background): “Stacked LSTM (Graves, 2013) 1.67 GF-LSTM (Chung et al., 2015) 1.58 Grid-LSTM (Kalchbrenner et al., 2016a) 1.47 Layer-normalized LSTM (Chung et al., 2016a) 1.46 MI-LSTM (Wu et al., 2016b) 1.44 Recurrent Memory Array Structures (Rocki, 2016) 1.40 HM-LSTM (Chung et al., 2016a) 1.40 Layer Norm HyperLSTM”
Character-Level Language Modeling with Deeper Self-Attention (2018, background): “Many authors explore RNN cell designs, such as Multiplicative Integration [46], MuFuRU [43], and Hypernetworks [13], including automatically searching for superior RNN cell designs [52].”
Learning Transferable Visual Models From Natural Language Supervision (2021, background): “…this way, the image encoder is the computer vision backbone which computes a feature representation for the image and the text encoder is a hypernetwork (Ha et al., 2016) which generates the weights of a linear classifier based on the text specifying the visual concepts that the classes represent.”
Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “There are two categories of meta-learning: 1. learning a meta-policy for updating model parameters (see, e.g., Andrychowicz et al., 2016; Ha et al., 2016a; Mishra et al., 2017) 2. learning a good parameter initialization for fast adaptation (see, e.g., Finn et al., 2017; Vinyals et al., 2016; Snell…”
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, methodology): “This approach has been applied to learning to optimize deep networks (Hochreiter et al., 2001; Andrychowicz et al., 2016; Li & Malik, 2017), as well as for learning dynamically changing recurrent networks (Ha et al., 2017).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “…sharing small network modules, for example the attention parameters (Firat et al., 2016a), and sharing everything except the attention parameters (Ha et al., 2016b; Blackwood et al., 2018), to sharing all parameters in the model across all language pairs (Johnson et al., 2017; Ha et al., 2016c).”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, methodology): “This precedent led many to use character-level language modeling as a benchmark to evaluate neural architectures (Kalchbrenner et al., 2016; Chung et al., 2017; Ha et al., 2017; Zilly et al., 2017; Melis et al., 2018; Al-Rfou et al., 2019).”
Neural Ordinary Differential Equations (2018, background): “This is parameterization is a kind of hypernetwork (Ha et al., 2016).”
GAN Inversion: A Survey (2021, background): “[66] further introduce a hypernetwork [102] that learns to refine the generator weights with respect to a given input image.”
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, background): “This kind of function can serve either as a regularizer during training or a hypernetwork that directly outputs model parameters or, equivalently, some task representation (Ha et al., 2017).”
CrossTransformers: spatially-aware few-shot transfer (2020, background): “The meta-learner either learns to produce new parameters directly from the new data [9, 33, 56, 62, 64, 72, 73], or learns to produce an update rule to iteratively optimize the base learner to fit the new data [2, 6, 8, 38, 63, 97].”
Meta-Learning Probabilistic Inference for Prediction (2018, methodology): “This is an example usage of hyper-networks (Ha et al., 2016) to amortize learning about weights, and can be recovered by the ML-PIP framework by pre-training θ and performing MAP inference for ψ .”
Fast Model Editing at Scale (2021, methodology): “…strategies for model editing exist, including modiﬁcations of standard ﬁne-tuning intended to enforce locality by reducing distance traveled in parameter space (Zhu et al., 2020) or even ﬁnd the min-L2 2021)) learn to edit a base model through meta-learning (Finn et al., 2017; Ha et al., 2017).”
Editing Factual Knowledge in Language Models (2021, neutral): “Concretely, KNOWLEDGEEDITOR is a hypernetwork (Ha et al., 2016)—i.”
Dynamic Neural Networks: A Survey (2021, background): “Dynamic filter networks (DFN) [116] and HyperNetworks [117] are two classic approaches realizing runtime weight prediction for CNNs and RNNs, respectively.”
Recurrent World Models Facilitate Policy Evolution (2018, methodology): “Future work will explore replacing the VAE and MDN-RNN with higher capacity models [27, 89, 93, 97, 98], or incorporating an external memory module [19, 107], if we want our agent to learn to explore more complicated worlds.”
FiLM: Visual Reasoning with a General Conditioning Layer (2017, neutral): “For example, FiLM can be viewed as using one network to generate parameters of another network, making it a form of hypernetwork (Ha, Dai, and Le 2016).”
Neural Architecture Search with Reinforcement Learning (2016, methodology): “We use an experimental setup that is similar to Ha et al. (2016), but use variational dropout by Gal (2015).”