Network Pruning via Transformable Architecture Search

By Xuanyi Dong, Yi Yang. 2019

271 citations recorded by the source corpus.

Network pruning reduces the computation costs of an over-parameterized network without performance damage. Prevailing pruning algorithms pre-define the width and depth of the pruned networks, and then transfer parameters from the unpruned network to pruned networks. To break the structure limitation of the pruned networks, we propose to apply neural architecture search to search directly for a network with flexible channel and layer sizes. The number of the channels/layers is learned by minimizing the loss of the pruned networks. The feature map of the pruned network is an aggregation of K feature map fragments (generated by K networks of different sizes), which are sampled based on the probability distribution.The loss can be back-propagated not only to the network weights, but also to the parameterized distribution to explicitly tune the size of the channels/layers. Specifically, we apply channel-wise interpolation to keep the feature map with different channel sizes aligned in the aggregation procedure. The maximum probability for the size in each distribution serves as the width and depth of the pruned network, whose parameters are learned by knowledge transfer, e.g., knowledge distillation, from the original networks. Experiments on CIFAR-10, CIFAR-100 and ImageNet demonstrate the effectiveness of our new perspective of network pruning compared to traditional network pruning algorithms. Various searching and knowledge transfer approaches are conducted to show the effectiveness of the two components. Code is at: this https URL.

Subfields: Computer Networks and Communications, Software, Efficiency & Compression, Neural Architecture Search & AutoML, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Software

In Efficiency & Compression

In Neural Architecture Search & AutoML

In Signal Processing

Citations in the corpus

AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, background): “2% Neural Architecture Search (NAS) has brought significant improvements in many applications, such as machine perception [19, 48, 5, 51, 50], language modeling [32, 11], and model compression [17, 10, 16].”
A Comprehensive Survey of Neural Architecture Search (2020, methodology): “With the improvement of NAS search efficiency, NAS is also quickly applied in the fields of object detection [65, 75, 111, 118], semantic segmentation [63, 64, 120], adversarial learning [53], architectural scaling [114, 122, 124], multi-objective optimization [39, 115, 125], platform-aware [28, 34,”
Differentiable Feature Aggregation Search for Knowledge Distillation (2020, background): “Therefore, the community has raised increasing attention on model compression approaches such as model pruning [7,21,34], model quantization [14,19,23] and knowledge distillation [13,30,33,37,38].”
NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size (2020, methodology): “For example, how to design a FLOPs constrain loss to regularize the discovered architecture to be efficient [21], [43], [44]?”
Learn-Prune-Share for Lifelong Learning (2020, unknown): “A Transformable Architecture Search (TAS) [23] realizes the pruned network and knowledge is transferred from the unpruned network to the pruned version.”
AutoHAS: Efficient Hyperparameter and Architecture Search. (2020, methodology): “Using a super model can avoid training millions of candidates from scratch (Liu et al., 2019; Dong & Yang, 2019a; Cai et al., 2019; Pham et al., 2018).”
Densely Connected Search Space for More Flexible Neural Architecture Search (2019, methodology): “ large scale dataset, i.e., ImageNet. FBNet [45] searches on the subset of ImageNet and uses the Gumbel Softmax function [23,35] to better optimize the distribution of architecture probabilities. TAS [13] utilizes a differentiable NAS scheme to search and prune the width and depth of the network and”
Teacher Supervises Students How to Learn From Partially Labeled Images for Facial Landmark Detection (2019, background): “Other teacher-student methods [38, 15, 10, 24] aim to fit the output of the student model to that of the teacher model.”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “TAS [182] exploits NAS to search for the depth and width of a network to obtain the pruned networks and uses knowledge distillation to train these pruned networks.”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “Transformable Architecture Search (TAS) [141] searches the width and depth of a network.”
Helpful or Harmful: Inter-task Association in Continual Learning (2022, methodology): “For the discrete optimization problem, we relax it as the following problem: To alleviate the latter problem, we present the Gumbel softmax trick [9,16] that produces the continuous random variable m i . where g i follows the Gumbel distribution [16] and τ is the control variable.”
AMER: Automatic Behavior Modeling and Interaction Exploration in Recommender System (2020, result): “Though diverse complex searching algorithms [5, 7, 10, 13] have been coupled with one-shot NAS, recent studies [4, 14, 37] reveal that the primitive random search with one-shot strategy is a surprisingly competitive baseline compared to gradient-based methods such as DARTS [44] and SNAS [76] or…”
One-Shot Neural Architecture Search via Self-Evaluated Template Network (2019, background): “Automatically discovering effective networks has attracted more and more researchers [6, 1, 42, 24, 15, 13].”