TAPAS: Train-less Accuracy Predictor for Architecture Search

By Roxana Istrate, Florian Scheidegger, Giovanni Mariani, Dimitrios S. Nikolopoulos, C. Bekas, A. Malossi. 2018

78 citations recorded by the source corpus.

In recent years an increasing number of researchers and practitioners have been suggesting algorithms for large-scale neural network architecture search: genetic algorithms, reinforcement learning, learning curve extrapolation, and accuracy predictors. None of them, however, demonstrated highperformance without training new experiments in the presence of unseen datasets. We propose a new deep neural network accuracy predictor, that estimates in fractions of a second classification performance for unseen input datasets, without training. In contrast to previously proposed approaches, our prediction is not only calibrated on the topological network information, but also on the characterization of the dataset-difficulty which allows us to re-tune the prediction without any training. Our predictor achieves a performance which exceeds 100 networks per second on a single GPU, thus creating the opportunity to perform large-scale architecture search within a few minutes. We present results of two searches performed in 400 seconds on a single GPU. Our best discovered networks reach 93.67% accuracy for CIFAR-10 and 81.01% for CIFAR-100, verified by training. These networks are performance competitive with other automatically discovered state-of-the-art networks however we only needed a small fraction of the time to solution and computational resources.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

A Constructive Prediction of the Generalization Error Across Scales (2019, background): “…at full data/model scale may be computationally prohibitive, a line of work tries to predict the performance of a given model on a given dataset, without training the model, for example by using a bank of previously trained models, dataset, and their associated performances (Istrate et al., 2019).”
Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor (2020, methodology): “In addition, the train-less accuracy predictor (TAP) [34] and the accelerating neural network architecture search (ANNAS) [35] algorithms also employed similar methods to predict the performance of the given CNNs.”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, background): “There are several works that predict the validation accuracy of neural networks (Klein et al. 2017; Deng, Yan, and Lin 2017; Istrate et al. 2019; Zhang, Ren, and Urtasun 2018; Baker et al. 2017).”
A Survey on Neural Architecture Search (2019, methodology): “ various hyperparameters. The parameters of the controller are updated with REINFORCE. In their work they explore transfer learning for both image classication and natural language processing tasks. Istrate et al. (2019) The approach by Istrate et al. (2019) is based on a surrogate model which predi”
Neural Architecture Search without Training (2020, methodology): “An orthogonal direction for identifying good architectures is the estimation of accuracy prior to training (Deng et al., 2017; Istrate et al., 2019), although these differ from this work in that they rely on training a predictive model, rather than investigating more fundamental architectural…”
NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search (2020, methodology): “Peephole (Deng et al., 2017) considers only forward architectures and aggregate the information about the operations in each layer using an LSTM. Similarly, TAPAS (Is-trate et al., 2019) uses an encoding like Peephole but extends the framework to deal with multiple datasets following the life-long l”
Fast Task-Aware Architecture Inference (2019, methodology): “The TAPAS system proposed in [10] also uses a history of past conﬁguration experiments stored in a database of experiments.”