Probabilistic Neural Architecture Search

By Francesco Paolo Casale, Jonathan Gordon, Nicoló Fusi. 2019

79 citations recorded by the source corpus.

In neural architecture search (NAS), the space of neural network architectures is automatically explored to maximize predictive accuracy for a given task. Despite the success of recent approaches, most existing methods cannot be directly applied to large scale problems because of their prohibitive computational complexity or high memory usage. In this work, we propose a Probabilistic approach to neural ARchitecture SEarCh (PARSEC) that drastically reduces memory requirements while maintaining state-of-the-art computational complexity, making it possible to directly search over more complex architectures and larger datasets. Our approach only requires as much memory as is needed to train a single architecture from our search space. This is due to a memory-efficient sampling procedure wherein we learn a probability distribution over high-performing neural network architectures. Importantly, this framework enables us to transfer the distribution of architectures learnt on smaller problems to larger ones, further reducing the computational cost. We showcase the advantages of our approach in applications to CIFAR-10 and ImageNet, where our approach outperforms methods with double its computational cost and matches the performance of methods with costs that are three orders of magnitude larger.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

In Neural Architecture Search & AutoML

Citations in the corpus

Understanding and Robustifying Differentiable Architecture Search (2019, background): “A recent line of work focuses on relaxing the neural architecture search space by making discrete decisions continuous and thus learnable via gradient descent [Liu et al., 2019, Xie et al., 2019, Casale et al., 2019, Cai et al., 2019].”
PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2019, methodology): “8% over PARSEC).”
A Survey on Neural Architecture Search (2019, methodology): “Casale et al. (2019) learn the parameters of an independent categorical distribution to sample architectures.”
XNAS: Neural Architecture Search with Expert Advice (2019, methodology): “For example [25, 49, 6, 3] used adam [20], and [29] used SGD with momentum.”
DATA: Differentiable ArchiTecture Approximation With Distribution Guided Sampling (2020, background): “Benefiting from such efforts, searching cost is reduced remarkably and significant successes have been achieved in a multitude of fields, including image classification [31], [32], [33], [34], semantic segmentation [35], [36], and object detection [37], [38], [39], [40].”
PC-DARTS: Partial Channel Connections for Memory-Efficient Differentiable Architecture Search (2019, methodology): “PARSEC [5] also proposed a sampling-based optimization method to learn a probability distribution.”
NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search (2020, background): “Subsequent work on DARTS has addressed further lowering the computational and the memory requirements (Dong & Yang, 2019; Xu et al., 2020; Cai et al., 2019; Casale et al., 2019).”
One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, unknown): “5 Whole Supernet Gradient PARSEC [8] 2.”
ASAP: Architecture Search, Anneal and Prune (2019, unknown): “…3150 NASNet-A (Zoph et al., 2018) 2.65 3.3 1800 PNAS (Liu et al., 2018a) 3.41 3.2 150 SNAS (Xie et al., 2019) 2.85 ± 0.02 2.8 1.5 DSO-NAS (Zhang et al., 2018) 2.95 ± 0.12 3 1 PARSEC (Casale et al., 2019) 2.81 ± 0.03 3.7 1 DARTS(2nd) (Liu et al., 2018b) 2.76 ± 0.06 3.4 1 PC-DARTS (Laube, 2019) 2 .”
Probabilistic Dual Network Architecture Search on Graphs (2020, background): “First, RL requires a full train and evaluate cycle for each architecture that is considered; making it computational expensive (Casale et al., 2019).”
Overcoming Multi-Model Forgetting in One-Shot NAS With Diversity Maximization (2020, unknown): “5 Whole Supernet Gradient PARSEC [8] 2.”
DATA: Differentiable ArchiTecture Approximation (2019, unknown): “, evolution-based NAS [13, 18, 26, 41, 43, 44, 45, 47], reinforcement learning-based NAS [2, 3, 21, 42, 56, 59, 60], and gradient-based NAS [11, 34, 35, 46, 53], which has achieved significant successes in a multitude of fields, including image classification [4, 12, 21, 30, 31, 34, 44, 53, 60], sem”