AMC: AutoML for Model Compression and Acceleration on Mobile Devices

By Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, Song Han. 2018

1,502 citations recorded by the source corpus.

Model compression is an effective technique to efficiently deploy neural network models on mobile devices which have limited computation resources and tight power budgets. Conventional model compression techniques rely on hand-crafted features and require domain experts to explore the large design space trading off among model size, speed, and accuracy, which is usually sub-optimal and time-consuming. In this paper, we propose AutoML for Model Compression (AMC) which leverages reinforcement learning to efficiently sample the design space and can improve the model compression quality. We achieved state-of-the-art model compression results in a fully automated way without any human efforts. Under 4\(\times \) FLOPs reduction, we achieved 2.7% better accuracy than the hand-crafted model compression method for VGG-16 on ImageNet. We applied this automated, push-the-button compression pipeline to MobileNet-V1 and achieved a speedup of 1.53\(\times \) on the GPU (Titan Xp) and 1.95\(\times \) on an Android phone (Google Pixel 1), with negligible loss of accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, background): “Model compression (Han et al., 2016; He et al., 2018; Yang et al., 2018) is a common way to reduce model size by trading accuracy for efﬁciency.”
The State of Sparsity in Deep Neural Networks (2019, methodology): “Liu et al. (2018) do explore convolutional architectures on the Ima-geNet datasets, but only at two relatively low sparsity levels (30% and 60%).”
MnasNet: Platform-Aware Neural Architecture Search for Mobile (2018, methodology): “Some commonly-used approaches include 1) quantizing the weights and/or activations of a baseline CNN model into lower-bit representations [8, 16], or 2) pruning less important ﬁlters according to FLOPs [6, 10], or to platform-aware metrics such as latency introduced in [32].”
AutoML: A Survey of the State-of-the-Art (2019, background): “approaches have been proposed to compress the neural networks into more efficient forms [161, 162, 163].”
Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, neutral): “Our work also relates to efficient deep learning on edge devices, such as model compression [43, 44, 45], manually designed architectures (MobileNets [46], ShuffeNets [47], SqueezeNets [48]), or even efficient neural architecture search (EfficientNets [4], FBNet [49]).”
VEGA: Towards an End-to-End Configurable AutoML Pipeline (2020, methodology): “A complete AutoML system should integrate necessary modules and functions such as NAS, HPO, data augmentation, and model compression (He et al., 2018).”
AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, neutral): “The resulting framework, dubbed AutoGAN-Distiller ( AGD ), is the first AutoML framework dedicated to GAN compression (alongside the concurrent work (Li et al., 2020)), and is also among a few earliest works that explore AutoML for GANs (Gong et al., 2019).”
AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, background): “< 56.2% Neural Architecture Search (NAS) has brought signiﬁcant improvements in many applications, such as machine perception [19, 48, 5, 51, 50], language modeling [32, 11], and model compression [17, 10, 16]. Most NAS works apply the same hyper-parameters while searching for network architectur”
Once for All: Train One Network and Specialize it for Efficient Deployment (2019, methodology): “…very effective technique for efﬁcient deep learning, including network pruning that removes redundant units (Han et al., 2015) or redundant channels (He et al., 2018; Liu et al., 2017), and quantization that reduces the bit width for the weights and activations (Han et al., 2016; Courbariaux…”
AutoGAN: Neural Architecture Search for Generative Adversarial Networks (2019, neutral): “It is also worth mentioning that NAS is also applied to CNN compression and acceleration [17].”
Network Pruning via Transformable Architecture Search (2019, methodology): “, pruning 30% filters in each layer [27, 15], predicting sparsity ratio [12], or leveraging regularization [2].”
A Survey on Neural Architecture Search (2019, methodology): “He et al. (2018) In their work, He et al. (2018) use a reinforcement controller to prune a trained network in a layer-wise fashion.”
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search (2018, background): “Besides searching for new architectures, works such as [27, 5] focus on adapting existing models to improve efficiency.”
Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, methodology): “We randomly sample 3k images from the training set , which follows (He et al., 2018b), to evaluate the objective in (9) for each candidate.”
Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search (2018, methodology): “More recently, Liu et al. (2018) propose DARTS, a differentiable architecture search framework.”
Single Path One-Shot Neural Architecture Search with Uniform Sampling (2019, background): “Previous works usually deal with the problem by channel pruning [8, 9] or model adaption [30, 6].”
Rethinking the Value of Network Pruning (2018, methodology): “[25] Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han.”