NVIDIA Tensor Core Programmability, Performance & Precision

By Stefano Markidis, Steven W. D. Chien, Erwin Laure, Ivy Bo Peng, Jeffrey S. Vetter. IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum, 2018

474 citations recorded by the source corpus.

The NVIDIA Volta GPU microarchitecture introduces a specialized unit, called Tensor Core that performs one matrix-multiply-and-accumulate on 4x4 matrices per clock cycle. The NVIDIA Tesla V100 accelerator, featuring the Volta microarchitecture, provides 640 Tensor Cores with a theoretical peak performance of 125 Tflops/s in mixed precision. In this paper, we investigate current approaches to program NVIDIA Tensor Cores, their performances and the precision loss due to computation in mixed precision. Currently, NVIDIA provides three different ways of programming matrix-multiply-and-accumulate on Tensor Cores: the CUDA Warp Matrix Multiply Accumulate (WMMA) API, CUTLASS, a templated library based on WMMA, and cuBLAS GEMM. After experimenting with different approaches, we found that NVIDIA Tensor Cores can deliver up to 83 Tflops/s in mixed precision on a Tesla V100 GPU, seven and three times the performance in single and half precision respectively. A WMMA implementation of batched GEMM reaches a performance of 4 Tflops/s. While precision loss due to matrix multiplication with half precision input might be critical in many HPC applications, it can be considerably reduced at the cost of increased computation. Our results indicate that HPC applications using matrix multiplications can strongly benefit from using of NVIDIA Tensor Cores.

Subfields: Computational Mathematics, Computational Mechanics, Signal Processing

View Paper: https://www.semanticscholar.org/paper/c904d42a6aeb81c385dbc94d95380734dee43fbc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mathematics

In Computational Mechanics

In Signal Processing

Citations in the corpus

DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction (2022, background): “…frequency, FP16 embedding with stochastic rounding, BF16 optimizer, and quantized all to all and allreduce collectives [44, 47] to further reduce memory footprint, help with numeric stability, leverage specialized accelerator hardware such as Tensor Core [24] and to reduce communication overhead.”
This paper is included in the Proceedings of the 2022 USENIX Annual Technical Conference. (unknown): “Markidis et al. [44] perform a performance evaluation of TC in mixed precision as well as three different approaches of programming matrix-multiply-and-accumulate on TC on V100.”
MLPerf Training Benchmark (2019, background): “As the number of hardware and software systems for DL training increases (Paszke et al., 2017; Abadi et al., 2016; Chen et al., 2015; Jia et al., 2014; Jouppi et al., 2017; Chen et al., 2018; Markidis et al., 2018; Intel, 2019), so does the need for a comprehensive benchmark.”
Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale (2021, methodology): “NVIDIA Tensor Cores support mixed precision of fp32 and fp16 to deliver 7x more ﬂops/s than fp32 on a Tesla V100 GPU [42].”
On the use of simulation in robotics: Opportunities, challenges, and suggestions for moving forward (2020, background): “For instance, recent graphics processing unit (GPU) cards can manage the simultaneous execution of more than 150,000 computing threads (16).”
Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks (2020, unknown): “Several studies have developed techniques such as loss scaling, stochastic rounding, and others to train effectively in 16-bit (Micikevicius et al., 2018; Das et al., 2018; Azim), along with associated hardware support (Markidis et al., 2018).”
Federated Learning in Edge Computing: A Systematic Survey (2022, methodology): “TensorFlow Federated (TFF): TFF [108] [116] is an FL benchmarking system that has applications in FL, multi-task learning, meta-learning, and on-device learning.”
Machine Learning and Deep Learning frameworks and libraries for large-scale data mining: a survey (2019, background): “Again this speeds up algebra calculation as well as greatly decreasing memory consumption at the cost of a slightly less accurate model (Iandola et al. 2016; Markidis et al. 2018).”
Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark (2018, methodology): “We suspect that the entries that use the V100s do not fully use the Tensor Cores, which are advertised to deliver 125 Teraflops of half-precision arithmetic [33].”
Mixed Precision Training With 8-bit Floating Point (2019, background): “Today most of the neural network training in a production deployment has migrated to 16-bit hardware, resulting in signiﬁcant improvements[20] in performance.”
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping (2020, methodology): “However, it requires Tensor Cores [19], which do not exist in all hardware.”
nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models (2022, methodology): “Another issue with running GPUs for inference is that GPUs have prioritized high memory bandwidth over memory size [31], [32].”
Sponge Examples: Energy-Latency Attacks on Neural Networks (2020, background): “Intel’s Knights Mill CPU provides a set of SIMD instructions [13], while NVIDIA’s Volta GPU introduces Tensor Cores to facilitate the low-precision multiplications that underpin much of deep learning [14].”
TurboTransformers: an efficient GPU serving system for transformer models (2020, methodology): “The turbo-TC allows GEMM operation to use Tensor Core [18] if possible.”
QKD: Quantization-aware Knowledge Distillation (2019, unknown): “[28, 33, 40, 42].”