More is Less: A More Complicated Network with Less Inference Complexity

By Xuanyi Dong, Junshi Huang, Yi Yang, Shuicheng Yan. 2017

302 citations recorded by the source corpus.

In this paper, we present a novel and general network structure towards accelerating the inference process of convolutional neural networks, which is more complicated in network structure yet with less inference complexity. The core idea is to equip each original convolutional layer with another low-cost collaborative layer (LCCL), and the element-wise multiplication of the ReLU outputs of these two parallel layers produces the layer-wise output. The combined layer is potentially more discriminative than the original convolutional layer, and its inference is faster for two reasons: 1) the zero cells of the LCCL feature maps will remain zero after element-wise multiplication, and thus it is safe to skip the calculation of the corresponding high-cost convolution in the original convolutional layer, 2) LCCL is very fast if it is implemented as a 1*1 convolution or only a single filter shared by all channels. Extensive experiments on the CIFAR-10, CIFAR-100 and ILSCRC-2012 benchmarks show that our proposed network structure can accelerate the inference process by 32% on average with negligible performance drop.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

What is the State of Neural Network Pruning? (2020, methodology): “As others have noted (Lebedev et al., 2014; Figurnov et al., 2016; Louizos et al., 2017; Yang et al., 2017; Han et al., 2015; Kim et al., 2015; Wen et al., 2016; Luo et al., 2017; He et al., 2018b), these metrics are far from perfect.”
Network Pruning via Transformable Architecture Search (2019, methodology): “Unstructured pruning methods [26, 7, 5, 9] usually enforce the convolutional weights [26, 11] or feature maps [5, 7] to be sparse.”
AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, background): “Extensive works [20, 19, 34, 12, 18, 17] have been done on accelerating neural networks by compression.”
Learning to Prune Filters in Convolutional Neural Networks (2018, methodology): “Similarly, [7] used low-cost collaborative kernels for acceleration.”
Bayesian Compression for Deep Learning (2017, methodology): “Inspired by this observation, several authors proposed pruning schemes that take these considerations into account [Wen et al., 2016, Yang et al., 2017] or even go as far as efﬁciency aware architectures to begin with [Iandola et al., 2017, Dong et al., 2017, Howard et al., 2017].”
SkipNet: Learning Dynamic Routing in Convolutional Networks (2017, background): “ng the gating modules) parameterized by " and g "{0,1}N . The overall objective is deÞned as min J (")=min E x E g L ! (g ,x ) =min E x E g # L (öy (x ,F ! ,g ),y) ! # N $N i=1 R i % , (5) where R i =(1! gi)C i is the reward of each gating module. The constant C i is the cost of execu”
Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks (2018, methodology): “We compare the performance with other state-of-the-art acceleration algorithms, e.g., [Dong et al. , 2017a; Li et al. , 2017; He et al. , 2017; Luo et al. , 2017].”
Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling (2018, background): “CNN becomes popular recently on some NLP tasks because of its the highly parallelizable convolution computation (Dong et al., 2017).”
Compressing Neural Networks using the Variational Information Bottleneck (2018, methodology): “…networks with low computational cost and/or memory footprint, prior work has involved designing more eﬃcient architectures (Howard et al., 2017; Dong et al., 2017; Iandola et al., 2016), quantizing network weights (Courbariaux et al., 2016, 2015; Han et al., 2015a; Mellempudi et al., 2017;…”
Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough (2020, unknown): “79G More is Less (Dong et al., 2017) 73.”
Pruning Filter via Geometric Median for Deep Convolutional Neural Networks Acceleration (2018, methodology): “, MIL [4], PFEC [15], CP [13], ThiNet [17], SFP [12], NISP [29], Rethinking [28].”
Dual Dynamic Inference: Enabling More Efficient, Adaptive, and Controllable Deep Inference (2019, methodology): “State-of-the-Art Techniques on CIFAR-10: We compare IADI against six state-of-the-art techniques including four dynamic skipping techniques (SkipNet [10], BlockDrop [34], SACT and ACT [8]) and two static compression techniques (PFEC [24] and LCCL [45]).”
Provable Filter Pruning for Efficient Neural Networks (2019, unknown): “80 Dong et al. (2017) 30.”
ADC: Automated Deep Compression and Acceleration with Reinforcement Learning (2018, background): “Since the development of optimal brain damage [35, 21], extensive work has been done on CNN compression [20, 19, 40, 15].”
Supervision-by-Registration: An Unsupervised Approach to Improve the Precision of Facial Landmark Detectors (2018, methodology): “The first detector is CPM [35], which utilizes the ImageNet pre-trained models [31, 7, 10] as the feature extraction part.”
Learning Instance-wise Sparsity for Accelerating Deep Models (2019, methodology): “[Dong et al., 2017] used low-cost collaborative layer(LCCL) to pre-compute the non-zero elements in the out feature maps, then ignored the locations with zeros.”
Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling (2018, background): “, 2015] or convolutional [Dong et al., 2017] neural networks on NLP tasks, such as machine translation [Vaswani et al.”