Learnable Embedding Space for Efficient Neural Architecture Compression

By Shengcao Cao, Xiaofang Wang, Kris M. Kitani. 2019

45 citations recorded by the source corpus.

We propose a method to incrementally learn an embedding space over the domain of network architectures, to enable the careful selection of architectures for evaluation during compressed architecture search. Given a teacher network, we search for a compressed network architecture by using Bayesian Optimization (BO) with a kernel function defined over our proposed embedding space to select architectures for evaluation. We demonstrate that our search algorithm can significantly outperform various baseline methods, such as random search and reinforcement learning (Ashok et al., 2018). The compressed architectures found by our method are also better than the state-of-the-art manually-designed compact architecture ShuffleNet (Zhang et al., 2018). We also demonstrate that the learned embedding space can be transferred to new settings for architecture search, such as a larger teacher network or a teacher network in a different architecture family, without any training. Code is publicly available here: this https URL .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Comprehensive Survey of Neural Architecture Search (2020, background): “…also quickly applied in the fields of object detection [65, 75, 111, 118], semantic segmentation [63, 64, 120], adversarial learning [53], architectural scaling [114, 122, 124], multi-objective optimization [39, 115, 125], platform-aware [28, 34, 103, 117], data augmentation [121, 123] and so on.”
A Survey on Neural Architecture Search (2019, methodology): “Cao et al. (2019) Cao et al. (2019) maximize Equation (25) by using Bayesian optimization (Section 3.3) with a Gaussian process built using a deep kernel (Wilson et al., 2016).”
Random Search and Reproducibility for Neural Architecture Search (2019, background): “[8] Shengcao Cao, Xiaofang Wang, and Kris M.”
AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification (2020, background): “Various search algorithms have been explored in NAS, such as random search [13,37], reinforcement learning [1,41,42,39], evolutionary algorithms [34,22,21], Bayesian optimization (BO) [11,3], and differentiable methods [16].”
Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints (2019, unknown): “[37] and neural architecture search [74, 7, 40].”
NAT: Neural Architecture Transformer for Accurate and Compact Architectures (2019, unknown): “propose an automatic architecture compression method [5].”