Peephole: Predicting Network Performance Before Training

By Boyang Deng, Junjie Yan, Dahua Lin. 2017

118 citations recorded by the source corpus.

The quest for performant networks has been a significant force that drives the advancements of deep learning in recent years. While rewarding, improving network design has never been an easy journey. The large design space combined with the tremendous cost required for network training poses a major obstacle to this endeavor. In this work, we propose a new approach to this problem, namely, predicting the performance of a network before training, based on its architecture. Specifically, we develop a unified way to encode individual layers into vectors and bring them together to form an integrated description via LSTM. Taking advantage of the recurrent network's strong expressive power, this method can reliably predict the performances of various network architectures. Our empirical studies showed that it not only achieved accurate predictions but also produced consistent rankings across datasets -- a key desideratum in performance prediction.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quoc V. Le
2. Mohit Bansal
3. Frank Hutter
4. Quanming Yao
5. Yoshua Bengio
6. Jianfeng Gao
7. Dacheng Tao
8. Percy Liang

In Evaluation & Benchmarks

1. Frank Hutter
2. Mengye Ren
3. Quoc V. Le
4. Amos J. Storkey
5. Barret Zoph
6. Mengjie Zhang
7. Bing Xue
8. Colin White

In Neural Architecture Search & AutoML

1. Arber Zela
2. Colin White
3. Yash Savani
4. Quoc V. Le
5. Frank Hutter
6. Quanming Yao
7. Xiaojun Chang
8. Xuanyi Dong

Citations in the corpus

• SNAS: Stochastic Neural Architecture Search (2018, background): “Ideas to design auxiliary mechanisms like performance prediction (Baker et al., 2017; Deng et al., 2017), iterative search (Liu et al.”
• Neural Architecture Optimization (2018, neutral): “It is noteworthy here that the regression model is similar to the performance predictor in previous works [4, 26, 10].”
• A Comprehensive Survey of Neural Architecture Search (2020, methodology): “NAO [73] uses a performance predictor similar to previous work [20, 36, 109].”
• Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor (2020, methodology): “output channels of each block varies between [32, 512] that is set based on the conventions of state-of-the-art CNNs [50], [54].”
• BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “[6] Boyang Deng, Junjie Yan, and Dahua Lin.”
• Graph HyperNetworks for Neural Architecture Search (2018, background): “Baker et al. (2017b) propose directly predicting performance from the learning curve, and Deng et al. (2017) propose to predict performance directly from the architecture without learning curve information.”
• TAPAS: Train-less Accuracy Predictor for Architecture Search (2018, methodology): “Compared to Peephole [4], TAP leverages knowledge accumulated through experiments of datasets of similar difficulty filtered from the LDE based on the DCN.”
• Neural Predictor for Neural Architecture Search (2019, methodology): “Unlike above methods, which used ν-SVR/random forests [1], multi-layer perceptrons [13], LSTMs [15, 7] or Gaussian processes [6], we use a Graph Convolutional Network (GCN) [10] for our regression model.”
• Neural Architecture Search without Training (2020, methodology): “An orthogonal direction for identifying good architectures is the estimation of accuracy prior to training (Deng et al., 2017; Istrate et al., 2019), although these differ from this work in that they rely on training a predictive model, rather than investigating more fundamental architectural…”
• Efficient Neural Architecture Search via Parameter Sharing (2018, background): “ENAS’s design of sharing weights between architectures is inspired by the concept of weight inheritance in neural model evolution (Real et al., 2017; 2018).”
• NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search (2020, methodology): “Peephole (Deng et al., 2017) considers only forward architectures and aggregate the information about the operations in each layer using an LSTM. Similarly, TAPAS (Is-trate et al., 2019) uses an encoding like Peephole but extends the framework to deal with multiple datasets following the life-long…”
• Fast Task-Aware Architecture Inference (2019, unknown): “• Performance prediction methods [10, 4, 2, 12].”
• Taking Human out of Learning Applications: A Survey on Automated Machine Learning (2018, background): “, [7], [20], [40], [54], [55], [56], [57], [58], [59], [60], [61], [62], [72]).”