Benchmarking State-of-the-Art Deep Learning Software Tools

By Shaohuai Shi, Qiang Wang, Pengfei Xu, Xiaowen Chu. 2016

343 citations recorded by the source corpus.

Deep learning has been shown as a successful machine learning method for a variety of tasks, and its popularity results in numerous open-source deep learning software tools coming to public. Training a deep network is usually a very time-consuming process. To address the huge computational challenge in deep learning, many tools exploit hardware features such as multi-core CPUs and many-core GPUs to shorten the training and inference time. However, different tools exhibit different features and running performance when they train different types of deep networks on different hardware platforms, making it difficult for end users to select an appropriate pair of software and hardware. In this paper, we present our attempt to benchmark several state-of-the-art GPU-accelerated deep learning software tools, including Caffe, CNTK, TensorFlow, and Torch. We focus on evaluating the running time performance (i.e., speed) of these tools with three popular types of neural networks on two representative CPU platforms and three representative GPU platforms. Our contribution is two-fold. First, for end users of deep learning software tools, our benchmarking results can serve as a reference to selecting appropriate hardware platforms and software tools. Second, for developers of deep learning software tools, our in-depth analysis points out possible future directions to further optimize the running performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

Citations in the corpus

Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, methodology): “Another benchmarking is provided in [53], comparing the running performance of Caffe, TensorFlow, Torch, CNTK, and MXNet.”
Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark (2018, background): “Many prior ML benchmarks use throughput (either per-kernel or per-iteration) as a metric [6,11,12,21,30,58].”
Deep Learning in Mining Biological Data (2020, background): “) in CPU and GPU platforms. The numbers in (a, c) denote the number of CPU threads employed in the benchmarking process, and in (d-f) denote the batch size. In case of DA the batch size was 64. times [155,156]. Moreover, to the best of our knowledge, there exist two main efforts that provide the ben”
A Survey of Deep Learning: Platforms, Applications and Emerging Research Trends (2018, methodology): “[119] showed that TensorFlow performs best in server-grade multi-thread (more than 8) implementations.”
Benchmarking TPU, GPU, and CPU Platforms for Deep Learning (2019, result): “compare CPU (Intel i73820 and E5-2630v3) and GPU (GTX 980, GTX 1080, and K80) platforms and deep learning frameworks [48].”
Deep learning for smart fish farming: applications, opportunities and challenges (2020, background): “Another limitation is that DL models demand high levels of computing power; in fact the available common CPUs are typically unable to meet the requirements of DL (Shi et al., 2016).”
A Survey on Distributed Machine Learning (2019, background): “However, the direct comparison of this number to the other synchronous frameworks’ results is questionable, as the dependency structure of an LSTM is signicantly dierent than that of an ordinary DNN due to the introduction of temporal state [140].”
DAWNBench : An End-to-End Deep Learning Benchmark and Competition (2017, background): “Most existing benchmarks for deep learning performance [1, 2, 10], and some research papers [4, 5, 8, 35], only measure proxy metrics such as the time to process one minibatch of data.”
Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community (2017, background): “The survey paper by Shi et al.60 investigates tools performance to help users of common DL tools (see Sec.”
An Introduction to Neural Information Retrieval (2018, unknown): “Refer to [Shi et al., 2016] for a comparison of different neural toolkits based on their speed of training using standard performance benchmarks.”
Distributed deep learning on edge-devices: Feasibility via adaptive compression (2017, background): “Since DNN models are large (from thousands to billions parameters [26]), placing those worker tasks over edge-devices imply significant updates transfer over the Internet.”