Weight Agnostic Neural Networks

By Adam Gaier, David R Ha. 2019

262 citations recorded by the source corpus.

Not all neural network architectures are created equal, some perform much better than others for certain tasks. But how important are the weight parameters of a neural network compared to its architecture? In this work, we question to what extent neural network architectures alone, without learning any weight parameters, can encode solutions for a given task. We propose a search method for neural network architectures that can already perform a task without any explicit weight training. To evaluate these networks, we populate the connections with a single shared weight parameter sampled from a uniform random distribution, and measure the expected performance. We demonstrate that our method can find minimal neural network architectures that can perform several reinforcement learning tasks without weight training. On a supervised learning domain, we find network architectures that achieve much higher than chance accuracy on MNIST using random weights. Interactive version of this paper at this https URL

Subfields: Optimization & Training, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Neural Architecture Search & AutoML

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “Another interesting paper recently published, involving Weight Agnostic Neural Networks [170], claims that neural network architectures alone, without learning any weight parameters, may encode solutions for a given task.”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “Other specific architecture aspects have also been targeted, such as the hyperparameters [77, 50, 32, 44], activation functions [66], a specific layer [35], the full forward pass [23], the data augmentation [17, 62, 18], etc.”
Observational Overfitting in Reinforcement Learning (2019, background): “…works which avoid overparametrization on simplifying policies (Rajeswaran et al., 2017; Mania et al., 2018) or compactifying networks (Choromanski et al., 2018; Gaier & Ha, 2019), we instead ﬁnd that there are generalization beneﬁts to overparametrization even in the nonlinear control case.”
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020, background): “Recent work (Gaier and Ha, 2019; Zhou et al., 2019) points out the potential of searching neural architectures within a ﬁxed model, as an alternative to directly optimizing the model weights for downstream tasks.”
Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks (2020, background): “…of individual weights has been studied in network pruning (LeCun et al., 1990; Hassibi & Stork, 1993; Li et al., 2017; Frankle & Carbin, 2019; Gaier & Ha, 2019) and feature attribution (Simonyan et al., 2013; Springenberg et al., 2015; Sundararajan et al., 2017; Shrikumar et al., 2017) often…”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “Inspired by Weight Agnostic Neural Networks ( WANNs ) [77], Ramanujan et al. [78] empirically ﬁnd that with an un-trained network grows wider and deeper, it will contain a sub-network that performs as well as a trained network with the same number of parameters.”
Continual Learning with Deep Artificial Neurons (2020, background): “ation of this synaptic vector. Weight-Agnostic Neural Networks demonstrated that there exist architectures, not speciﬁc weight conﬁgurations, which exhibit out-of-the-box propensity for certain tasks [8]. The Lottery Ticket Hypothesis has shown that there often exist smaller, easily optimizable netw”
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future (2020, methodology): “Recent work using random weights, for example, helps to isolate the role of architecture versus specific weight values (Gaier & Ha, 2019).”
Deeper Insights into Weight Sharing in Neural Architecture Search (2020, background): “Population based algorithms (Gaier & Ha, 2019; Liang et al., 2018; Jaderberg et al., 2017) are another popular approach for NAS, new trials could inherit neural architecture from better performing ones as well as their weights, and mutate the architecture to explore better ones.”
What’s Hidden in a Randomly Weighted Neural Network? (2019, background): “In Weight Agnostic Neural Networks (WANNs) [6], Gaier and Ha question if an architecture alone may encode the solution to a problem.”
Neuroevolution of self-interpretable agents (2020, background): “solution to a small subspace of all possible solutions offered by direct encoding methods. This constraint naturally incorporates into our agent an inductive bias that determines what it does well at [29, 42, 103], and this bias is dependent on the choice of our indirect encoding method. arXiv:2003.”
Rethinking and Improving the Robustness of Image Style Transfer (2021, background): “For example, Gaier et al. [5] pro-pose the weight agnostic neural network, by ﬁxing randomly initialized weights and searching optimal network architectures, which is shown to achieve good results in several reinforcement and supervised learning tasks.”
What shapes feature representations? Exploring datasets, architectures, and training (2020, background): “Evolved architectures with untrained weights have been successful in actionplanning problems [9].”
Learning to Predict Without Looking Ahead: World Models Without Forward Prediction (2019, methodology): “We used the pre-trained VAE made available in [16] with a latent size of 16 which trained following the same procedure as [22].”