Convolutional neural networks at constrained time cost

By Kaiming He. 2014

1,384 citations recorded by the source corpus.

Though recent advanced convolutional neural networks (CNNs) have been improving the image recognition accuracy, the models are getting more complex and time-consuming. For real-world applications in industrial and commercial scenarios, engineers and developers are often faced with the requirement of constrained time budget. In this paper, we investigate the accuracy of CNNs under constrained time cost. Under this constraint, the designs of the network architectures should exhibit as trade-offs among the factors like depth, numbers of filters, filter sizes, etc. With a series of controlled comparisons, we progressively modify a baseline model while preserving its time complexity. This is also helpful for understanding the importance of the factors in network designs. We present an architecture that achieves very competitive accuracy in the ImageNet dataset (11.8% top-5 error, 10-view test), yet is 20% faster than “AlexNet” [14] (16.0% top-5 error, 10-view test).

Subfields: Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Deep Residual Learning for Image Recognition (2015, neutral): “Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments.”
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, methodology): “In [11], aggressively increasing the depth leads to saturated or degraded accuracy.”
On Network Design Spaces for Visual Recognition (2019, methodology): “From prior work we know that a model’s error is related to its complexity; in particular more complex models are often more accurate [7, 34, 11, 39].”
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017, background): “The increasing needs of running high quality deep neural networks on embedded devices encourage the study on efficient model designs [8].”
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size (2016, methodology): “Sun applied delayed downsampling to four different DNN architectures, and in each case delayed downsampling led to higher classification accuracy [10].”
Pruning Filters for Efficient ConvNets (2016, methodology): “Additionally, quantization (Han et al. (2016b)) and binarization (Rastegari et al. (2016); Courbariaux & Bengio (2016)) can be used to reduce the model size and lower the computation overheads.”
PolyNet: A Pursuit of Structural Diversity in Very Deep Networks (2016, background): “In addition, extensive studies [6, 7, 26, 17] have been done to find good design choices, e.”
Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs (2017, neutral): “Therefore, the restriction of ECC to 1-hop neighborhoods N(i) is not a constraint, akin to using small 3×3 filters in normal CNNs in exchange for deeper networks, which is known to be beneficial [17].”
Fast ConvNets Using Group-Wise Brain Damage (2015, background): “It is interesting to see that the optimization process decided to shrink the receptive fields towards the center compared to the full version (which is consistent with the findings in [43, 19]).”
Accelerating Very Deep Convolutional Networks for Classification and Detection (2015, background): “It may not be beneficial to uniformly accelerate all layers.”
Practical Network Blocks Design with Q-Learning (2017, methodology): “We will also try to search blocks with limited parameters and FLOPs, and try to conduct experiments on other tasks such as detection or segmentation.”
Learning Deep ResNet Blocks Sequentially using Boosting Theory (2017, background): “pirical studies suggest that challenging tasks in image classification (He et al., 2015; Ioffe & Szegedy, 2015; Simonyan & Zisserman, 2014; Szegedy et al., 2015) and object recognition (Girshick, 2015; Girshick et al.”
PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions (2015, background): “Another direction of research is modifications of CNN architecture for speed gains [12, 15, 25].”
BlockQNN: Efficient Block-Wise Neural Network Architecture Generation (2018, background): “Meanwhile, we notice that the FLOPs and density of the corresponding blocks have a negative correlation with the ﬁnal accuracy.”
INet: Convolutional Networks for Biomedical Image Segmentation (2021, background): “The computational complexity of a single Conv-layer [55] can be measured by: where M is the side length of each convolution kernel’s output feature map.”
- LEVEL ACCURACY WITH 50 X FEWER PARAMETERS AND < 0 . 5 MB MODEL SIZE (2016, methodology): “He and H. Sun applied delayed downsampling to four different CNN architectures, and in each case delayed downsampling led to higher classification accuracy (He & Sun, 2015).”
Attention Residual Learning for Skin Lesion Classification (2019, background): “However, due to degradation problems [41], [42], it is still”
Deep Residual Networks with Exponential Linear Unit (2016, background): “The training error increases as we add more layers to a deep model, as mentioned in [2].”