Benchmark Analysis of Representative Deep Neural Network Architectures

By Simone Bianco, Rémi Cadène, Luigi Celona, Paolo Napoletano. 2018

739 citations recorded by the source corpus.

This paper presents an in-depth analysis of the majority of the deep neural networks (DNNs) proposed in the state of the art for image recognition. For each DNN, multiple performance indices are observed, such as recognition accuracy, model complexity, computational complexity, memory usage, and inference time. The behavior of such performance indices and some combinations of them are analyzed and discussed. To measure the indices, we experiment the use of DNNs on two different computer architectures, a workstation equipped with a NVIDIA Titan X Pascal, and an embedded system based on a NVIDIA Jetson TX1 board. This experimentation allows a direct comparison between DNNs running on machines with very different computational capacities. This paper is useful for researchers to have a complete view of what solutions have been explored so far and in which research directions are worth exploring in the future, and for practitioners to select the DNN architecture(s) that better fit the resource constraints of practical deployments and applications. To complete this work, all the DNNs, as well as the software used for the analysis, are available online.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Evaluation & Benchmarks

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “Model Our Profile (ms) [7] (ms)”
What is the State of Neural Network Pruning? (2020, methodology): “Results for non-pruned models are taken from (Tan & Le, 2019) and (Bianco et al., 2018).”
High Quality Monocular Depth Estimation via Transfer Learning (2018, methodology): “We have experimented with different state-of-the-art encoders [2], of more or less complexity than that of DenseNet-169, and we also looked at different decoder types [23, 36].”
Taming Visually Guided Sound Generation (2021, background): “We attribute the small difference between the RGB+flow features and RGB-only features to the fact that ResNet50 is a stronger architecture than BN-Inception on the ImageNet benchmark [3].”
Large-Scale Representation Learning from Visually Grounded Untranscribed Speech (2019, background): “See Bianco et al. (2018) for an extensive benchmark analysis of popular convolutional neural network architectures.”
MLPerf Inference Benchmark (2019, background): “An example of ML-model diversity for image classification (plot from [9]).”
Efficient Attention Network: Accelerate Attention by Searching Where to Plug (2020, background): “The operations and trainable modules in the implementation of self-attention inevitably require extra computational cost and parameters, resulting in slow inference and cumbersome network [2].”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “Figure 7 show top-1 accuracy percent verses the number of operations needed for a number of popular neural networks [23].”