DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures

By Jin-Dong Dong, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, Min Sun. 2018

197 citations recorded by the source corpus.

Recent breakthroughs in Neural Architectural Search (NAS) have achieved state-of-the-art performances in applications such as image classification and language modeling. However, these techniques typically ignore device-related objectives such as inference time, memory usage, and power consumption. Optimizing neural architecture for device-related objectives is immensely crucial for deploying deep networks on portable devices with limited computing resources. We propose DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures, optimizing for both device-related (e.g., inference time and memory usage) and device-agnostic (e.g., accuracy and model size) objectives. DPP-Net employs a compact search space inspired by current state-of-the-art mobile CNNs, and further improves search efficiency by adopting progressive search (Liu et al. 2017). Experimental results on CIFAR-10 are poised to demonstrate the effectiveness of Pareto-optimal networks found by DPP-Net, for three different devices: (1) a workstation with Titan X GPU, (2) NVIDIA Jetson TX1 embedded system, and (3) mobile phone with ARM Cortex-A53. Compared to CondenseNet and NASNet (Mobile), DPP-Net achieves better performances: higher accuracy & shorter inference time on various devices. Additional experimental results show that models found by DPP-Net also achieve considerably-good performance on ImageNet as well.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Neural Architecture Search & AutoML

Citations in the corpus

Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “However, also for larger models, LEMONADE is competitive to methods that require signiﬁcantly more computational resources (Zoph et al., 2018) or start their search with non-trivial architectures (Cai et al., 2018b; Dong et al., 2018).”
A Comprehensive Survey of Neural Architecture Search (2020, methodology): “5, the curved dotted line indicates the dense connection in Dpp-net [34].”
A Survey on Neural Architecture Search (2019, background): “Dong et al. (2018) propose a search space specifically aimed at meeting such requirements which include objectives like fewer parameters and fast inference time.”
Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours (2019, methodology): “Hardware-aware NAS: Earlier NAS methods focused on maximizing accuracy under FLOPs constraints [22,25], but low FLOP count does not necessarily translate to hardware efficiency [10,18].”
Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, background): “Lastly, (Dong et al., 2018) proposed to learn a surrogate function that predicts the accuracy of network candidates, which makes the traversal of the Pareto frontier efﬁcient.”
Graph HyperNetworks for Neural Architecture Search (2018, methodology): “While earlier NAS methods focused on standard image classiﬁcation and language modeling, recent literature has extended NAS to search for architectures that are computationally efﬁcient (Tan et al., 2018; Dong et al., 2018; Hsu et al., 2018; Elsken et al., 2018; Zhou et al., 2018).”
Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search (2019, methodology): “Other searching tactics like sequential model-based optimization (SMBO) are also exhibited in DeepArchitect (Negrinho & Gordon, 2017), PNAS (Liu et al., 2017a), and in PPP-Net (Dong et al., 2018).”
Densely Connected Search Space for More Flexible Neural Architecture Search (2019, unknown): “Then evolutionary algorithm (EA) based meth-ods [12, 30, 39] achieve great performance.”
AMER: Automatic Behavior Modeling and Interaction Exploration in Recommender System (2020, methodology): “In the second step, AMER introduces sequential model-based optimization (SMBO) algorithm [12, 43] to search for low-order and high-order interactions on non-sequential features by progressively increasing the order of interactions.”
RC-DARTS: Resource Constrained Differentiable Architecture Search (2019, methodology): “Compared with DPPNet, as shown in Table 2, on Cifar10, RC-DARTS-C14 largely reduces the error rate of DPPNet by 1.67% (a 28.6% relative improvements).”
Partial Order Pruning: For Best Speed/Accuracy Trade-Off in Neural Architecture Search (2019, background): “Despite a few works [8, 2, 28] consider the actual inference speed on target platforms, they search the architecture in each individual building block and keep fixed the overall architecture, i.”
MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning (2019, methodology): “AutoML Recently, AutoML methods [21, 52, 8, 12] take the real-time inference latency on multiple devices into account to iteratively prune channels in different layers of a network via reinforcement learning [21] or an automatic feedback loop [52].”
Learnable Embedding Space for Efficient Neural Architecture Compression (2019, background): “Some existing works in NAS are focused on searching for architectures that not only can achieve high performance but also respect some resource or computation constraints (Ashok et al., 2018; Tan et al., 2018; Zhou et al., 2018; Dong et al., 2018; Hsu et al., 2018; Elsken et al., 2018a).”