Predicting Parameters in Deep Learning

By Misha Denil, Babak Shakibi, Laurent Dinh, Marc'Aurelio Ranzato, Nando de Freitas. 2013

1,410 citations recorded by the source corpus.

We demonstrate that there is significant redundancy in the parameterization of several deep learning models. Given only a few weight values for each feature it is possible to accurately predict the remaining values. Moreover, we show that not only can the parameter values be predicted, but many of them need not be learned at all. We train several different architectures by learning only a small number of weights and predicting the rest. In the best case we are able to predict more than 95% of the weights of a network without any drop in accuracy.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Factorization tricks for LSTM networks (2017, background): “Denil et al. (2013) have shown that it is possible to train several different deep architectures by learning only a small number of weights and predicting the rest.”
Going deeper with convolutions (2014, methodology): “We show incremental domain expansion is effective in applying object detectors, trained with only ImageNet, to videos, improving performance by approximately 48% on Activities of Daily Living (ADL) dataset [3] and by 15% on the YouTube Objects dataset [4].”
Federated Learning: Strategies for Improving Communication Efficiency (2016, neutral): “Fixing Bt (randomly or deterministically) is also an option, but our approach seems to perform as well as the best techniques considered in [6] without any need of hand-crafted features.”
HyperNetworks (2016, background): “…concept of a network interacting with another network is central to the work of (Jaderberg et al., 2016; Andrychowicz et al., 2016), and especially (Denil et al., 2013; Yang et al., 2015; Bertinetto et al., 2016; De Brabandere et al., 2016), where certain parameters in a convolutional network are…”
FastText.zip: Compressing text classification models (2016, positive): “Denil et al. (2013) show that such classiﬁcation models are easily compressed because they are over-parametrized, which concurs with early observations by LeCun et al. (1990).”
Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Some methods approximate model weights using quantization and hashing [Gong et al., 2014, Courbariaux et al., 2015], or low-rank approximation and tensor factorization [Denil et al., 2013, Lebedev et al., 2015, Su et al., 2018].”
LoRA: Low-Rank Adaptation of Large Language Models (2021, background): “Some prior works even explicitly impose the low-rank constraint when training the original neural network (Sainath et al., 2013; Povey et al., 2018; Zhang et al., 2014; Jaderberg et al., 2014; Zhao et al., 2016; Khodak et al., 2021; Denil et al., 2014); however, to the best of our knowledge, none of”
Initialization and Regularization of Factorized Neural Layers (2021, background): “When these are smaller than the original matrix, the resulting networks are more efficient for both training and inference (Denil et al., 2013; Moczulski et al., 2015; Ioannou et al., 2016; Tai et al., 2016), resulting in model compression.”
Measuring the Intrinsic Dimension of Objective Landscapes (2018, methodology): “( ii ) Unlike layerwise compression models (Denil et al., 2013; Wen et al., 2016), we operate in the entire parameter space, which could work better or worse, depending on the network.”
Speeding up Convolutional Neural Networks with Low Rank Expansions (2014, background): “While a few other CNN acceleration methods exist, our key insight is to exploit the redundancy that exists between different feature channels and filters [6].”
Learning feed-forward one-shot learners (2016, background): “The general idea of predicting parameters has been explored before by Denil et al. [3], who showed that it is possible to linearly predict as much as 95% of the parameters in a layer given the remaining 5%.”
Dynamic Neural Networks: A Survey (2021, background): “1), weight prediction [108] is more straightforward: it directly generates (a subset of) instancewise parameters with an independent model at test time (see”
Learning both Weights and Connections for Efficient Neural Network (2015, background): “Neural networks are typically over-parameterized, and there is significant redundancy for deep learning models [10].”
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015, background): “Neural networks are typically over-parametrized, and there is signiﬁcant redundancy for deep learning models(Denil et al., 2013).”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “Denil et al. (2013) represent weight matrices as products of lower-rank factors.”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, background): “This typically results in ineﬃcient computation and memory usage [12].”
Pruning Filters for Efficient ConvNets (2016, methodology): “To reduce the computation costs of the convolutional layers, past work have proposed to approximate convolutional operations by representing the weight matrix as a low rank product of two smaller matrices without changing the original number of filters (Denil et al. (2013); Jaderberg et al. (2014); ”
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation (2014, background): “While these networks tend to be hugely over parameterized [2], this redundancy seems necessary in order to overcome a highly nonconvex optimization [3].”
The Loss Surfaces of Multilayer Networks (2014, background): “It was recently shown [Denil et al., 2013, Denton et al., 2014] that for large-size networks large number of network parameters (according to [Denil et al., 2013] even up to 95”