Differentiable Neural Network Architecture Search

By Richard Shin, Charles Packer, Dawn Song. 2018

94 citations recorded by the source corpus.

Subfields: Optimization & Training, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Neural Architecture Search & AutoML

Citations in the corpus

AutoML: A Survey of the State-of-the-Art (2019, methodology): “The widely used methods contains reinforcement learning (RL) [5], [6], [8], [9], [14], evolutionary algorithm (EA) [15]–[21], and gradient descent (GD) [10], [22], [23], Bayesian optimization [24]–[30] and so on.”
A Comprehensive Survey of Neural Architecture Search (2020, background): “12, 31], EA [15, 42], Bayesian optimization [61], SMBO [36] and MCTS [62] is that they regard network architecture search as a black-box optimization problem in a discrete search space. For this, DAS [68] explores to transform the discrete network architecture space into a continuously differentiabl”
NAS-Unet: Neural Architecture Search for Medical Image Segmentation (2019, methodology): “The search algorithms mainly includes heuristic algorithm [19], [21], [32]–[34], reinforcement learning [35]–[38], [38], [39], the Bayesian optimization method [40], [41] and the gradient-based method [20], [42], [43].”
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation (2019, methodology): “Image classiﬁcation is a good starting point for NAS, because it is the most fundamental and well-studied high-level recognition task.”
DATA: Differentiable ArchiTecture Approximation With Distribution Guided Sampling (2020, methodology): “zation problem, gradient-based NAS methods utilize gradients to optimize neural architecture [27], [28], [29], [30].”
Fast Task-Aware Architecture Inference (2019, background): “Previous work [13, 18] has shown that relaxing the parametrization from discrete to continuous space allows for eﬃcient gradient-based optimization schemes while still providing competitive model performances.”
DARTS: Differentiable Architecture Search (2018, background): “The idea of searching architectures within a continuous domain is not new (Saxena and Verbeek, 2016; Ahmed and Torresani, 2017; Shin et al., 2018), but there are several major distinctions.”
Differentiable Architecture Search with Ensemble Gumbel-Softmax (2019, methodology): “Contrary to treating architecture search as black-box optimization problem, gradient based neural architecture search methods utilized the gradient obtained in the training process to optimize neural architecture (Shin et al., 2018; Luo et al., 2018; Liu et al., 2018b; Xie et al., 2018).”
Efficient Neural Architecture Search via Proximal Iterations (2019, background): “nly one operation will be selected. From the perspective of performance, operations typically correlate with each other [39, 12], e.g., a 7x7’s convolution ﬁlter can cover a 3x3 one as a special case [33]. When updating a network’s weights, the ensemble constructed by DARTS during the search may lea”
Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help? (2019, methodology): “widely used in computer vision in the early/mid 2000s [15]. First, we move away from black box optimization for architecture search, similar to strategies that have been adopted in other recent works [26]. Instead, we view the architecture as being composed of primitive basic blocks. The “skeleton” ”
Differentiable Neural Architecture Search via Proximal Iterations (2019, background): “, a 7x7’s convolution filter can cover a 3x3 one as a special case [33].”
Search to Distill: Pearls Are Everywhere but Not the Eyes (2019, background): “search [19,22], evolutionary search [28], and other algorithms [1]. Recently, some works have also started to explore simultaneous optimization of network parameters and architecture for a given task [31]. It has been shown that NAS can discover novel architectures that surpass human-designed archit”
SGAS: Sequential Greedy Architecture Search (2019, background): “sands of GPU hours. One shot approaches [6,5,8] further reduce the search time by training a single over-parameterized network with inherited/shared weights. In order to search in a continuous domain [44,1,46,53], DARTS [35] proposes a continuous relaxation of the architecture representation, making”
Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization (2019, methodology): “While concurrent methods consider relaxed convolution formulations with insight similar to our work [33], [34], [35], [21], they either use design spaces and objectives that have been shown to be hardware inefﬁcient ( e.g. , cell-based space, FLOP count), or they optimize over a subset of our…”
DATA: Differentiable ArchiTecture Approximation (2019, unknown): “, evolution-based NAS [13, 18, 26, 41, 43, 44, 45, 47], reinforcement learning-based NAS [2, 3, 21, 42, 56, 59, 60], and gradient-based NAS [11, 34, 35, 46, 53], which has achieved significant successes in a multitude of fields, including image classification [4, 12, 21, 30, 31, 34, 44, 53, 60], sem”