An Analysis of Deep Neural Network Models for Practical Applications

By Alfredo Canziani, Adam Paszke, Eugenio Culurciello. 2016

1,252 citations recorded by the source corpus.

Since the emergence of Deep Neural Networks (DNNs) as a prominent technique in the field of computer vision, the ImageNet classification challenge has played a major role in advancing the state-of-the-art. While accuracy figures have steadily increased, the resource utilisation of winning models has not been properly taken into account. In this work, we present a comprehensive analysis of important metrics in practical applications: accuracy, memory footprint, parameters, operations count, inference time and power consumption. Key findings are: (1) power consumption is independent of batch size and architecture; (2) accuracy and inference time are in a hyperbolic relationship; (3) energy constraint are an upper bound on the maximum achievable accuracy and model complexity; (4) the number of operations is a reliable estimate of the inference time. We believe our analysis provides a compelling set of information that helps design and engineer efficient DNNs.

Subfields: Artificial Intelligence, Building and Construction, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Building and Construction

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors (2016, background): “, [3]), detection models tend to be significantly more complex.”
The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning (2017, background): “, (Canziani et al., 2016) for a summary of different architectures).”
Reconciling modern machine-learning practice and the classical bias–variance trade-off (2018, methodology): “Thus, for instance, data sets as large as ImageNet [33], which has ∼ 10 6 examples and ∼ 10 3 classes, may require networks with ∼ 10 9 parameters to achieve interpolation; this is larger than many neural network models for ImageNet [11].”
Energy and Policy Considerations for Deep Learning in NLP (2019, methodology): “nergy use required fortraining and inference in popular convolutional models for image classiﬁcation in computer vision, including ﬁne-grained analysis comparing different neural network layer types. Canziani et al. (2016) assess image classiﬁcation model accuracy as a function of model size and gig”
Green AI (2019, background): “Third, FPO is strongly correlated with the running time of the model [5].”
Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, background): “[174] analyzed several state-of-the-art DNN models to find out the relationship between their accuracy, memory usage, operations count, inference time, and power consumption.”
AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, neutral): “We empirically observe that Error is inversely-proportional to log(FLOPs) or log(#Param) [7].”
Financial Trading Model with Stock Bar Chart Image Time Series with Deep Convolutional Neural Networks (2019, background): “Following these improvements, diﬀerent versions of Inception (v3, v4) and ResNet (v50, v101, v152) algorithms were proposed for image classiﬁcation and their success rates were approximately 75-80% [2].”
Deep Meta-Learning: Learning to Learn in the Concept Space (2018, methodology): “According to the analysis of (Canziani et al., 2016), we choose ResNet-50 (He et al., 2016) (excluding the last layer) as our concept generator in the experiments.”
PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning (2017, background): “The results presented so far were obtained for the vanilla VGG-16 network, a simple and large network, well known to be full of redundancies [2].”
Generative Poisoning Attack Method Against Neural Networks (2017, background): “For example, a small-footprint large vocabulary speech recognizer for mobile devices [12], a small robust deep-learning model designed to provide high quality text-to-speech functionality on smart devices [3], and an audio sensing framework built from coupled DNNs [4].”
Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization (2019, background): “However, forward pass time per batch item decreases with increasing batch size due to improved data parallelism [6], so it is important to compute costs with appropriate input dimensions.”
Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning (2020, background): “Similarly, the PFLOPs-hr metric can by multiplied by TDP (Watts) and divided by the maximum computational throughput of the GPU (in PFLOPs).”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, background): “[8] Alfredo Canziani, Adam Paszke, and Eugenio Culurciello.”
To understand deep learning we need to understand kernel learning (2018, background): “exceeding the size of training data by several orders of magnitude (Canziani et al., 2016).”