Improving the speed of neural networks on CPUs

By Vincent Vanhoucke, Andrew W. Senior, Mark Z. Mao. 2011

866 citations recorded by the source corpus.

Subfields: Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

In ML Systems & Hardware

Citations in the corpus

Speeding up Convolutional Neural Networks with Low Rank Expansions (2014, methodology): “[40] show that 8-bit quantization of the layer weights can result in a speedup with minimal loss of accuracy.”
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, background): “oates et al., 2013). As a result, it is often a challenge to run DNNs on target low-power devices, and substantial research eorts are invested in speeding up DNNs at run-time on both generalpurpose (Vanhoucke et al., 2011; Gong et al., 2014; Romero et al., 2014; Han et al., 2015b) and specialized co”
Learning both Weights and Connections for Efficient Neural Network (2015, background): “[11] explored a fixed-point implementation with 8-bit integer (vs 32-bit floating point) activations.”
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015, background): “There have been various proposals to remove the redundancy: Vanhoucke et al. (2011) explored a ﬁxed-point implementation with 8-bit integer (vs 32-bit ﬂoating point) activations.”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, methodology): “A ﬁxed-point implementation of 8-bit integer was compared with 32-bit ﬂoating point activations in [31].”
Deep Learning with Limited Numerical Precision (2015, methodology): “Some recent studies that embrace this approach have relied on the processor’s vector instructions to perform multiple 8 bit operations in parallel (Vanhoucke et al., 2011), or employ reconﬁgurable hardware (FPGAs) for high-throughput, energy-eﬃcient inference (Farabet et al., 2011; Gokhale et al.,…”
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation (2014, background): “[4] explored the properties of CPUs to speed up execution.”
Q8BERT: Quantized 8Bit BERT (2019, methodology): “In another example, Jacob et al. [5] introduced a method of training linear quantized Convolutional Neural Networks (CNN) that uses Integer Arithmetic instead of Floating Point Arithmetic which can be up to 4 × faster using only 25% of the memory footprint [13].”
Compressing Deep Convolutional Networks using Vector Quantization (2014, background): “Vanhoucke et al. (2011), who explored the properties of CPU to speed up the execution of CNN, particularly focused on the aligning of memory and SIMD operations to boost matrix operations.”
To prune, or not to prune: exploring the efficacy of pruning for model compression (2017, methodology): “Different quantization techniques such as ﬁxed-point quantization (Vanhoucke et al., 2011) or vector quantization (Gong et al., 2014) achieve different compression ratios and accuracies but also require different software or hardware to support inference at runtime.”
Fixed Point Quantization of Deep Convolutional Networks (2015, background): “This can be seen as a process of quantization.”
Opportunities and obstacles for deep learning in biology and medicine (2017, background): “The massively parallel nature of GPUs allows additional optimizations, such as accelerated mini-batch gradient descent [379,380,383,384].”
Training CNNs with Low-Rank Filters for Efficient Image Classification (2015, background): “For example Gupta et al. (2015) and Vanhoucke et al. (2011) achieve training- and test-time compute savings by further quantization of network weights that were originally represented as 32 bit floating point numbers.”
Efficient and accurate approximations of nonlinear convolutional networks (2014, background): “There have been a few studies on approximating deep CNNs for accelerating test-time evaluation [22, 3, 10].”
Binarized Neural Networks (2016, background): “…to run DNNs on target low-power devices, and substantial research efforts are invested in speeding up DNNs at run-time on both general-purpose (Vanhoucke et al., 2011; Gong et al., 2014; Romero et al., 2014; Han et al., 2015) and specialized computer hardware (Farabet et al., 2011a;b; Pham et…”
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients (2016, background): “In light of this, substantial research efforts are invested in speeding up DCNNs at both run-time and training-time, on both general-purpose (Vanhoucke et al., 2011; Gong et al., 2014; Han et al., 2015b) and specialized computer hardware (Farabet et al., 2011; Pham et al., 2012; Chen et al.,…”
Training with Quantization Noise for Extreme Model Compression (2020, methodology): “Popular postprocessing quantization methods, like scalar quantization, replace the floating-point weights of a trained network by a lower-precision representation, like fixed-width integers [7].”