Parameter Prediction for Unseen Deep Architectures
By Boris Knyazev, Michal Drozdzal, Graham W. Taylor, Adriana Romero-Soriano. 2021
109 citations recorded by the source corpus.
Deep learning has been successful in automating the design of features in machine learning pipelines. However, the algorithms optimizing neural network parameters remain largely hand-designed and computationally inefficient. We study if we can use deep learning to directly predict these parameters by exploiting the past knowledge of training other networks. We introduce a large-scale dataset of diverse computational graphs of neural architectures - DeepNets-1M - and use it to explore parameter prediction on CIFAR-10 and ImageNet. By leveraging advances in graph neural networks, we propose a hypernetwork that can predict performant parameters in a single forward pass taking a fraction of a second, even on a CPU. The proposed model achieves surprisingly good performance on unseen and diverse networks. For example, it is able to predict all 24 million parameters of a ResNet-50 achieving a 60% accuracy on CIFAR-10. On ImageNet, top-5 accuracy of some of our networks approaches 50%. Our task along with the model and results can potentially lead to a new, more computationally efficient paradigm of training networks. Our model also learns a strong representation of neural architectures enabling their analysis.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Alexei A. Efros
- Jitendra Malik
- Trevor Darrell
- Ilija Radosavovic
- Abhinav Kumar Gupta
- Luc Van Gool
- Sergey Levine
- William S. Peebles
In Computer Vision and Pattern Recognition
- Jitendra Malik
- Luc Van Gool
- Alexei A. Efros
- Trevor Darrell
- Ming-Hsuan Yang
- Ilija Radosavovic
- Dacheng Tao
- Andrew Zisserman
Citations in the corpus
- Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “…population-level trends (Radosavovic et al., 2019; 2020), benchmarking neural architecture search (Ying et al., 2019), training hypernetworks (Knyazev et al., 2021), predicting model properties (Sch¨urholt et al., 2021) and dataset distillation (Wang et al., 2018; Cazenavette et al., 2022).”