Graph HyperNetworks for Neural Architecture Search

By Chris Zhang, Mengye Ren, Raquel Urtasun. 2018

304 citations recorded by the source corpus.

Neural architecture search (NAS) automatically finds the best task-specific neural network topology, outperforming many manual architecture designs. However, it can be prohibitively expensive as the search requires training thousands of different networks, while each can last for hours. In this work, we propose the Graph HyperNetwork (GHN) to amortize the search cost: given an architecture, it directly generates the weights by running inference on a graph neural network. GHNs model the topology of an architecture and therefore can predict network performance more accurately than regular hypernetworks and premature early stopping. To perform NAS, we randomly sample architectures and use the validation accuracy of networks with GHN generated weights as the surrogate search signal. GHNs are fast -- they can search nearly 10 times faster than other random search methods on CIFAR-10 and ImageNet. GHNs can be further extended to the anytime prediction setting, where they have found networks with better speed-accuracy tradeoff than the state-of-the-art manual designs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neural Architecture Search & AutoML

Citations in the corpus

A Comprehensive Survey of Neural Architecture Search (2020, methodology): “Unlike SMASH, which encodes an architecture into a three-dimensional tensor through a memory channel scheme, Graph HyperNetwork (GHN) [90] recommends using computation graphs to represent the network architecture, and then using graph neural networks to perform architecture searches.”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “Another recent work uses a hypernetwork for neural network prediction in NAS [74].”
Memory-Efficient Hierarchical Neural Architecture Search for Image Denoising (2019, background): “Exemplar works include hyper-networks [40], network morphism [6] and shared weights [30].”
Searching for a Robust Neural Architecture in Four GPU Hours (2019, neutral): “54† GHN + CutOut [41] ICLR19 1 0.”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “Similar to other one-shot based NAS approaches (Bender et al., 2018; Zhang et al., 2019b; Liu et al., 2019b; Cai et al., 2019; Gordon et al., 2018), we also include (different or same) scaling scalars over all operations of all edges to control the information flow, denoted as woij which also…”
A Survey on Neural Architecture Search (2019, methodology): “Zhang et al. (2019) conduct a similar experiment using a separate implementation and confirm the correlation between the two variables.”
Random Search and Reproducibility for Neural Architecture Search (2019, background): “7 4 N/A N/A N gradient-based GHN#† [48] N/A 2.”
Automated Machine Learning: From Principles to Practices (2018, methodology): “There exist works in AutoML area [2, 110, 152] that make use of the idea of stochastic relaxation for search space transformation to handle the search problem.”
AutoDispNet: Improving Disparity Estimation With AutoML (2019, background): “Hypernetworks [10, 75] generate weights for candidate networks and evaluate them without training them until convergence.”
NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size (2020, methodology): “This can provide ground truth label for hypernetwork-based NAS methods [41], [42], which learn to generate parameters of an architecture.”
Parameter Prediction for Unseen Deep Architectures (2021, methodology): “The evaluation architectures with batch norm (BN) have running statistics, which are not learned by gradient descent [7], and (5)While source code for GHNs [24] is unavailable, we appreciate the authors’ help in implementing some steps.”
Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS (2020, background): “Each node represents an operation (such as a 1 × 1 convolution in CNN, and ReLU activation in LSTM) while edges represent data ﬂows [39].”
Network Slimming by Slimmable Networks: Towards One-Shot Architecture Search for Channel Numbers (2019, result): “Blue indicates the network pruning methods [27, 29, 30, 37, 38], Cyan indicates the network architecture search methods [25, 47, 48, 57] and Red indicates our results using AutoSlim.”
Few-shot Video-to-Video Synthesis (2019, background): “This class of networks has a different inductive bias to regular networks and has found use in several tasks including sequence modeling [15], image ﬁltering [23, 59, 49], frame interpolation [38, 37], and neural architecture search [64].”
Multi-objective Neural Architecture Search via Predictive Network Performance Optimization (2019, unknown): “Graph neural network has been proposed in previous work for predicting the parameters of the architecture using a graph hypernetwork (Zhang et al., 2019).”
A review of modularization techniques in artificial neural networks (2019, background): “One application which is very relevant to architectures and topologies is Graph HyperNetworks (Zhang et al. 2018).”
One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, background): “Recent studies have shown that NAS can significantly improve computational efficiency [3], [6], [63].”