Fast Convolutional Nets With fbfft: A GPU Performance Evaluation

By Nicolas Vasilache, Jeff Johnson, Michaël Mathieu, Soumith Chintala, Serkan Piantino, Yann LeCun. 2014

362 citations recorded by the source corpus.

We examine the performance profile of Convolutional Neural Network training on the current generation of NVIDIA Graphics Processing Units. We introduce two new Fast Fourier Transform convolution implementations: one based on NVIDIA's cuFFT library, and another based on a Facebook authored FFT implementation, fbfft, that provides significant speedups over cuFFT (over 1.5x) for whole CNNs. Both of these convolution implementations are available in open source, and are faster than NVIDIA's cuDNN implementation for many common convolutional layers (up to 23.5x for some synthetic kernel configurations). We discuss different performance regimes of convolutions, comparing areas where straightforward time domain convolutions outperform Fourier frequency domain convolutions. Details on algorithmic applications of NVIDIA GPU hardware specifics in the implementation of fbfft are also provided.

Subfields: Computer Vision and Pattern Recognition, Geophysics, Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Geophysics

In Efficiency & Compression

In ML Systems & Hardware

Citations in the corpus

Deep Convolutional Networks on Graph-Structured Data (2015, background): “Fourier implementations of Convnets [13, 19] bring the complexity to O ( N log N ) thanks again to the speciﬁc symmetries of the grid.”
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017, methodology): “Without modifying the parameters, optimized convolution algorithms implemented by FFT [26, 36] and other methods [2] decrease time consumption in practice.”
Recent advances in convolutional neural networks (2015, methodology): “testing process, such as cuDNN [75] and fbfft [76].”
AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, background): “10, 41] and special convolution implementations [36, 48, 29, 3] can also speed up the neural networks.”
Group Equivariant Convolutional Networks (2016, methodology): “Here we present the details for a G-convolution implementation that can leverage recent advances in fast computation of planar convolutions (Mathieu et al., 2014; Vasilache et al., 2015; Lavin & Gray, 2015).”
Spectral Representations for Convolutional Neural Networks (2015, background): “More recently, Mathieu et al. (2013); Vasilache et al. (2014) have demonstrated that convolution can be computed significantly faster using discrete Fourier transforms than directly in the spatial domain, even for tiny filters.”
Compressing Convolutional Neural Networks in the Frequency Domain (2015, methodology): “Another technique for speeding up convolutional neural network evaluation is computing convolutions in the Fourier frequency domain, as convolution in the spatial domain is equivalent to (comparatively lower-cost) element-wise multiplication in the frequency domain [38, 57].”
Fast ConvNets Using Group-Wise Brain Damage (2015, methodology): “In parallel to the lowering-based approaches mentioned above, which reduce convolutions to matrix multiplications, several works investigate the use of fast Fourier transforms [36, 44].”
Channel Pruning for Accelerating Very Deep Neural Networks (2017, methodology): “Recent CNN acceleration works fall into three categories: optimized implementation (e.g., FFT [48]), quantization (e.g., BinaryNet [8]), and structured simplification that convert a CNN into compact one [22].”
Fast Algorithms for Convolutional Neural Networks (2015, methodology): “Low performance at moderate values of N suggests that the FFT convolution implementation either uses large tiles, or possibly just a single tile per image, as in [12], which leads to inefficient multiplication stages unlessN is large.”
Scaling Local Self-Attention for Parameter Efficient Visual Backbones (2021, background): “Convolutions have improved over the last decade through changes in (a) the computation of the primitive [4, 25, 34, 52, 56, 28]; (b) the software implementation [5]; (c) the structure of the primitive itself, through for example, grouped convolution [57] and depthwise separable convolution [45].”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “utation, reducing the number of operations by up to 47%. A second method to compute convolutions is to make use of the Fourier domain, in which convolution is defined as an element-wise multiplication[166,232]. In this method, both the data and the kernels are transformed using FFT, multiplied, and ”
PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions (2015, neutral): “Recently, cufft library [31] has been introduced.”
Faster CNNs with Direct Sparse Convolutions and Guided Pruning (2016, background): “Winograd (Lavin & Gray, 2015) and FFT based algorithms (Vasilache et al., 2015) also aim to speedup convolution.”
Optimizing Performance of Recurrent Neural Networks on GPUs (2016, background): “There have been several studies optimizing the implementation of neural networks for GPUs, particularly in the case of convolutional neural networks [13, 14].”
DeepChain: Auditable and Privacy-Preserving Deep Learning with Blockchain-Based Incentive (2019, methodology): “To alleviate this problem, distributed deep learning training has been proposed recently, and some research work [43], [44], [45], [46], [47] and system implementations have been around, such as DistBelief [48], Torch [49], DeepImage [50] and Purine [51].”
Scalpel: Customizing DNN pruning to the underlying hardware parallelism (2017, positive): “[43] accelerate DNN by faster convolutional computation into FFT computation, which suffers from extra memory cost.”