MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks

By Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Tien-Ju Yang, Edward Choi. 2017

362 citations recorded by the source corpus.

We present MorphNet, an approach to automate the design of neural network structures. MorphNet iteratively shrinks and expands a network, shrinking via a resource-weighted sparsifying regularizer on activations and expanding via a uniform multiplicative factor on all layers. In contrast to previous approaches, our method is scalable to large networks, adaptable to specific resource constraints (e.g. the number of floating-point operations per inference), and capable of increasing the network's performance. When applied to standard network architectures on a wide variety of datasets, our approach discovers novel structures in each domain, obtaining higher performance while respecting the resource constraint.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, background): “Structure pruning [37, 18, 17, 27, 12, 24, 23, 7, 13, 10] and growing [9, 38, 27, 7] were proposed for different goals, such as efficient inference [37, 18, 17, 12, 24, 23, 7, 13, 10], lifelong learning [38] and model adaptation [9, 27].”
Green AI (2019, methodology): “Most of these approaches target ef-ﬁcient model inference [31, 49, 12], 15 and thus only minimize the cost of processing a single example, while ignoring the other two red practices discussed in Section 2.”
MnasNet: Platform-Aware Neural Architecture Search for Mobile (2018, methodology): “Some commonly-used approaches include 1) quantizing the weights and/or activations of a baseline CNN model into lower-bit representations [8, 16], or 2) pruning less important ﬁlters according to FLOPs [6, 10], or to platform-aware metrics such as latency introduced in [32].”
SNAS: Stochastic Neural Architecture Search (2018, background): “Rather than directly estimating the forwarding time, there are three candidates from the literature (Gordon et al., 2018; Ma et al., 2018) that can be used to approximately represent it: 1) the parameter size ; 2) the number of ﬂoat-point operations (FLOPs); and 3) the memory access cost (MAC).”
Network Pruning via Transformable Architecture Search (2019, background): “in different searching candidates is the same. TAS is a differentiable NAS method, which is able to efﬁciently search for a transformable networks with ﬂexible width and depth. Network transformation [5, 10, 3] also studied the depth and width of networks. Chen et al. [5] manually widen and deepen a”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “While SNAS applies a softened one-hot random variable to rank the architecture parameter, Gordon et al. (2018) treats the scaling factor of Batch Normalization as an edge and normalization as its associated operation.”
Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, methodology): “As a result, standard training procedures using stochastic gradient descent optimize both Θ and z jointly (Ye et al., 2018; Liu et al., 2017; Gordon et al., 2018).”
Rethinking the Value of Network Pruning (2018, methodology): “[24] Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi.”
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications (2018, methodology): “The figure compares NetAdapt (adapting the small MobileNetV1) with the multipliers [9] and MorphNet [5] on a mobile CPU of Google Pixel 1.”
ChamNet: Towards Efficient Network Design Through Platform-Aware Model Adaptation (2018, methodology): “MorphNet [9]: this is a network simplification algorithm based on sparsifying regularization.”
Importance Estimation for Neural Network Pruning (2019, methodology): “We compare against pruning methods that use various heuristics, such as weight magnitude [21, 23], magnitude of the batch-norm scale, BN scale [6, 24, 32], and output-based heuristics (Taylor expansion applied to layer outputs) [27].”
Designing Adaptive Neural Networks for Energy-Constrained Image Classification (2018, background): “Nevertheless, recent work argues that the effectiveness of methods targeting the weights and parameters of a monolithic CNN relies heavily on the performance of this baseline (seed) network [10].”
AutoPrune: Automatic Network Pruning by Regularizing Auxiliary Parameters (2019, methodology): “LeCun et al. [1990] pioneers neural network pruning and proposes optimal brain damage method for shallow neural network unstructured pruning.”
MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing (2019, methodology): “To address this limitation, we propose using a lasso regularization to automatically learn an architecture for our model (Gordon et al., 2018).”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “The size of the inference network executing on mobile devices may be limited due to memory, bandwidth, or processing constraints [73].”
Low-Memory Neural Network Training: A Technical Report (2019, methodology): “, 2018], and MorphNet [Gordon et al., 2018].”
Fast Sparse ConvNets (2019, neutral): “Due to the difficultly of accelerating sparse computation, channel pruning approaches have been preferred [10, 5, 29, 28, 42, 13].”