What’s Hidden in a Randomly Weighted Neural Network?

By Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, Mohammad Rastegari. 2019

414 citations recorded by the source corpus.

Training a neural network is synonymous with learning the values of the weights. By contrast, we demonstrate that randomly weighted neural networks contain subnetworks which achieve impressive performance without ever training the weight values. Hidden in a randomly weighted Wide ResNet-50 is a subnetwork (with random weights) that is smaller than, but matches the performance of a ResNet-34 trained on ImageNet. Not only do these ``untrained subnetworks" exist, but we provide an algorithm to effectively find them. We empirically show that as randomly weighted neural networks with fixed weights grow wider and deeper, an ``untrained subnetwork" approaches a network with learned weights in accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Theoretical Study on Solving Continual Learning (2022, methodology): “[79] Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari.”
Supermasks in Superposition (2020, methodology): “However, we initialize the real valued “scores” with Kaiming uniform as in [39].”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, background): “” This work is among several recent papers to propose that merely sparsifying at initialization can produce high performance neural networks (Mallya et al., 2018; Zhou et al., 2019; Ramanujan et al., 2020; Evci et al., 2020).”
Movement Pruning: Adaptive Sparsity by Fine-Tuning (2020, methodology): “Since the gradient of Top v is 0 everywhere it is deﬁned, we follow Ramanujan et al. [2020], Mallya and Lazebnik [2018] and approximate its value with the straight-through estimator [Bengio et al., 2013].”
Learning where to learn: Gradient sparsity in meta and continual learning (2021, methodology): “, 2013) that was recently used for similar large-scale masking (Ramanujan et al., 2020).”
What shapes feature representations? Exploring datasets, architectures, and training (2020, background): “Prior work has observed that the representations in untrained networks already transfer well to tasks like classification [17, 32], and denoising and inpainting ([40]).”
Proving the Lottery Ticket Hypothesis: Pruning is All You Need (2020, methodology): “We prove the latter conjecture, stated in [27], in the case of deep and shallow neural networks.”
Towards Automated Circuit Discovery for Mechanistic Interpretability (2023, background): “This is particularly relevant to work on subnetworks as empirically some subnetworks in models with randomized weights do not accurately represent such reasoning (Ramanujan et al., 2020).”
Provable and Efficient Continual Representation Learning (2022, methodology): “Inspired by [66], SupSup [84] trains a binary mask for each task while keeping the underlying model fixed at initialization.”
Forget-free Continual Learning with Winning Subnetworks (2022, methodology): “Since PackNet (Mallya & Lazebnik, 2018) includes layer-wise capacity c for parameters that will be isolated for an incoming new set of tasks, we fix c = 1 /τ while performing PMNIST and 5-Dataset experiments.”
Block Pruning For Faster Transformers (2021, methodology): “Score-based pruning methods (Ramanujan et al., 2019) modify the model by introducing score parameters S for each parameter i and replace the original parameter matrices with a masked version W′ = W M(S).”
VeRA: Vector-based Random Matrix Adaptation (2023, background): “In parallel to this, recent research has shown the surprising effectiveness of models utilizing random weights and projections (Peng et al., 2021; Ramanujan et al., 2020; Lu et al., 2022; Schrimpf et al., 2021; Frankle et al., 2021).”
Structured Pruning Adapters (2022, background): “In fact, a task can be “learned” simply by selecting the appropriate subset of weights in a network (Mallya et al., 2018; Ramanujan et al., 2020).”
Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection (2020, neutral): “Compared with weight pruning (e.g., Han et al., 2016b (Wang et al., 2019; Ramanujan et al., 2019).”
On the Soft-Subnetwork for Few-shot Class Incremental Learning (2022, methodology): “To update the weight scores, we use Straight-through Estimator [15, 2, 36] in the backward pass.”
Effective Sparsification of Neural Networks with Global Sparsity Constraint (2021, background): “2% remaining weights, while state-of-the-art method edgepopup [29] collapse with less than 30% accuracy.”
Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs (2020, background): “However, other papers have shown that it is possible to achieve high accuracy by learning a binarized mask in which individual weights at initialization are either retained or pruned (Zhou et al., 2019; Ramanujan et al., 2019).”
Open-World Continual Learning: Unifying Novelty Detection and Continual Learning (2023, methodology): “Two of our proposed CIL methods use two parameter-isolation methods (HAT [55] and SupSup [61]) for task incremental learning (TIL) as one of the components.”