maxDNN: An Efficient Convolution Kernel for Deep Learning with Maxwell GPUs

By Andrew Lavin. 2015

48 citations recorded by the source corpus.

This paper describes maxDNN, a computationally efficient convolution kernel for deep learning with the NVIDIA Maxwell GPU. maxDNN reaches 96.3% computational efficiency on typical deep learning network architectures. The design combines ideas from cuda-convnet2 with the Maxas SGEMM assembly code. We only address forward propagation (FPROP) operation of the network, but we believe that the same techniques used here will be effective for backward propagation (BPROP) as well.

Subfields: Artificial Intelligence, Oceanography, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Kurt Keutzer
2. Mohit Bansal
3. Luke S. Zettlemoyer
4. Quoc V. Le
5. Yoshua Bengio
6. Jianfeng Gao
7. Ali Ayub
8. Jan Kautz

In Oceanography

1. Timo Aila
2. Jan Kautz
3. Stephen Tyree
4. Pavlo Molchanov
5. Tero Karras
6. Scott Gray

In Efficiency & Compression

1. Kurt Keutzer
2. Pavlo Molchanov
3. Jan Kautz
4. Forrest N. Iandola
5. Stephen Tyree
6. Timo Aila
7. Tero Karras
8. Dmitry P. Vetrov

In Signal Processing

1. Tero Karras
2. Timo Aila
3. Jan Kautz
4. Stephen Tyree
5. Pavlo Molchanov
6. Scott Gray
7. Francisco Massa
8. Pushmeet Kohli

Citations in the corpus

• PyTorch: An Imperative Style, High-Performance Deep Learning Library (2019, background): “Specialized libraries such as cuDNN [22], along with a body of academic work (such as [23] and [24]), produced a set of high-performance reusable deep learning kernels that enabled frameworks such as Caffe [1], Torch7 [25], or TensorFlow [3] to take advantage of these hardware accelerators.”
• Fast Algorithms for Convolutional Neural Networks (2015, background): “Recent work has shown that the direct convolution algorithm for multi-channel filtering can be computed at near full utilization on modern computers.[8] Therefore further speedup requires fast algorithms.”
• Pruning Convolutional Neural Networks for Resource Efficient Transfer Learning (2016, methodology): “In this evaluation we focus on reducing the number of convolutional parameters and floating point operations (FLOPs) [3], assuming a sliding window implementation of the convolutions [8].”
• FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters (2015, background): “This is true even when leveraging deep neural network primitives like cuDNN [5], maxDNN [18], or fbfft [28] – all of which operate near the theoretical peak computation per second achievable on GPUs.”
• PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions (2015, background): “Implementations that exploit the parallelism available in computational architectures like GPUs (cuda-convnet2 [13], CuDNN [3]) have allowed to significantly reduce the evaluation time of CNNs.”
• Pruning Convolutional Neural Networks for Resource Efficient Inference (2016, methodology): “During pruning we were measuring reduction in computations by FLOPs, which is a common practice (Han et al., 2015; Lavin, 2015a;b).”
• Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking (2018, background): “Multiple authors [9, 13, 14, 15, 16, 17] have shown areas for improved efficiency in the machine code emitted by the NVIDIA CUDA Compiler (NVCC) for architectures preceding Volta.”