DeepArchitect: Automatically Designing and Training Deep Architectures

By Renato Negrinho, Geoffrey J. Gordon. 2017

194 citations recorded by the source corpus.

In deep learning, performance is strongly affected by the choice of architecture and hyperparameters. While there has been extensive work on automatic hyperpa- rameter optimization for simple spaces, complex spaces such as the space of deep architectures remain largely unexplored. As a result, the choice of architecture is done manually by the human expert through a slow trial and error process guided mainly by intuition. In this paper we describe a framework for automatically designing and training deep models. We propose an extensible and modular lan- guage that allows the human expert to compactly represent complex search spaces over architectures and their hyperparameters. The resulting search spaces are tree- structured and therefore easy to traverse. Models can be automatically compiled to computational graphs once values for all hyperparameters have been chosen. We can leverage the structure of the search space to introduce different model search algorithms, such as random search, Monte Carlo tree search (MCTS), and sequen- tial model-based optimization (SMBO). We present experiments comparing the different algorithms on CIFAR-10 and show that MCTS and SMBO outperform random search. We also present experiments on MNIST, showing that the same search space achieves near state-of-the-art performance with a few samples. These experiments show that our framework can be used effectively for model discov- ery, as it is possible to describe expressive search spaces and discover competitive models without much effort from the human expert. Code for our framework and experiments has been made publicly available

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Neural Architecture Search & AutoML

Citations in the corpus

VEGA: Towards an End-to-End Configurable AutoML Pipeline (2020, background): “DeepArchitect (Negrinho & Gordon, 2017) attempts to unify the representation of search spaces over architectures and their hyperparameters and then proposes an extensible and modular language.”
A Comprehensive Survey of Neural Architecture Search (2020, methodology): “An internal reason for the inefficiency of these mainstream search methods based on RL [11, 12, 31], EA [15, 42], Bayesian optimization [61], SMBO [36] and MCTS [62] is that they regard network architecture search as a black-box optimization problem in a discrete search space.”
Towards modular and programmable architecture search (2019, methodology): “Researchers have considered a variety of search algorithms, including reinforcement learning [13], evolutionary algorithms [14, 15], MCTS [16], SMBO [16, 17], and Bayesian optimization [18].”
Evolving Robust Neural Architectures to Defend from Adversarial Attacks (2019, methodology): “More- 2Code is available at http://bit.ly/RobustArchitectureSearch over, architectures compete with similar ones in their subpopulation, such that only the fittest of each subpopulation survives.”
Continual and Multi-Task Architecture Search (2019, methodology): “Some recent attempts have made architecture search more computationally feasible (Negrinho and Gordon, 2017; Baker et al., 2017) via tree-structured search space or Q-learning with an (cid:15) - greedy exploration, and further improvements via a weight-sharing strategy called Efﬁcient Neural…”
A Survey on Neural Architecture Search (2019, methodology): “The work of Negrinho and Gordon (2017) adopts the vanilla UCT algorithm with bisection for neural architecture search.”
Overcoming Multi-Model Forgetting (2019, neutral): “Initial search solutions required hundreds of GPUs due to the huge search space, but recent efforts have made the search more tractable, for example via the use of neural blocks (Negrinho & Gordon, 2017; Bennani-Smires et al., 2018).”
Random Search and Reproducibility for Neural Architecture Search (2019, background): “Since the work by Zoph and Le [49], there has been explosion of research activity on this problem [29, 30, 37, 11, 41, 1, 21, 5, 39, 48, 32, 45, 7].”
Evolutionary-Neural Hybrid Agents for Architecture Search (2018, neutral): “…deep RL and evolution, different approaches have been applied to architecture search and hyperparameter optimization: bayesian optimization [15], cascade-correlation [14], boosting [9], deep-learning based tree search [30], hill-climbing [13], model-based approaches [21], and random search [4].”
Efficient Progressive Neural Architecture Search (2018, background): “Surrogate aided exploration has also demonstrated accuracy and efficiency for network hyperparameter optimization [15, 17].”
Auto-Meta: Automated Gradient Based Meta Learner Search (2018, background): “This line of thinking is present in hypernetworks [4], co-evolving NEAT [16], boosting [6, 11], MCTS [18], early stopping of unpromising models [3], and progressive neural architecture search (PNAS) [14].”
Path-Level Network Transformation for Efficient Architecture Search (2018, neutral): “…categorized as neuro-evolution (Real et al., 2017; Liu et al., 2018), Bayesian optimization (Domhan et al., 2015; Men-doza et al., 2016), Monte Carlo Tree Search (Negrinho & Gordon, 2017) and reinforcement learning (RL) (Zoph & Le, 2017; Baker et al., 2017; Zhong et al., 2017; Zoph et al., 2017).”
Neural Architecture Search Using Deep Neural Networks and Monte Carlo Tree Search (2018, critical): “Monte Carlo Tree Search: DeepArchitect (Negrinho and Gordon 2017) implemented vanilla MCTS for NAS without a predictive model, and Wistuba (Wistuba 2017) uses statistics from the current search (e.g., RAVE and Contextual Re-ward Prediction) to predict the performance of a state.”
Transfer Learning with Neural AutoML (2018, methodology): “These include random search [1], parameter modeling [3], metalearned hyperparameter initialization [16], deep-learning based tree searches over a predefined model-specification language [17], and learning of gradient descent optimizers [18, 19].”
Neural Architecture Search with Bayesian Optimisation and Optimal Transport (2018, background): “A related line of work first sets up a search space for architectures via incremental modifications, and then explores this space via random exploration, MCTS, or A* search [6, 22, 29].”
Searching for Efficient Multi-Scale Architectures for Dense Image Prediction (2018, methodology): “Many parallel efforts have employed reinforcement learning [3, 99], evolutionary algorithms [81, 69, 59, 90, 53, 68] and sequential model-based optimization [61, 52] to learn network structures.”
DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures (2018, methodology): “[5] use Monte Carlo Tree Search (MCTS) to search through the space of CNN architectures in a shallow-to-deep manner and randomly select which branch to expand at each node.”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “Examples of SMBO include DeepArchitect [180], which proposes a DNN definition language that allows programmers to explicitly define the space; PNASNet [164], which searches for networks ordered by increasing complexity using a search algorithm based on A*, conserving half the evaluated models compar”