Dissecting the NVidia Turing T4 GPU via Microbenchmarking

By Zhe Jia, Marco Maggioni, Jeffrey K. Smith, Daniele Paolo Scarpazza. 2019

138 citations recorded by the source corpus.

In 2019, the rapid rate at which GPU manufacturers refresh their designs, coupled with their reluctance to disclose microarchitectural details, is still a hurdle for those software designers who want to extract the highest possible performance. Last year, these very reasons motivated us to dissect the Volta GPU architecture using microbenchmarks. The introduction in August 2018 of Turing, NVidia's latest architecture, pressed us to update our study. In this report, we examine Turing and compare it quantitatively against previous NVidia GPU generations. Specifically, we study the T4 GPU: a low-power board aiming at inference applications. We describe its improvements against its inference-oriented predecessor: the P4 GPU based on the Pascal architecture. Both T4 and P4 GPUs achieve significantly higher frequency-per-Watt figures than their full-size counterparts. We study the performance of the T4's TensorCores, finding a much higher throughput on low-precision operands than on the P4 GPU. We reveal that Turing introduces new instructions that express matrix math more succinctly. We map Turing's instruction space, finding the same encoding as Volta, and additional instructions. We reveal that the Turing TU104 chip has the same memory hierarchy depth as the Volta GV100; cache levels sizes on the TU104 are frequently twice as large as those found on the Pascal GP104. We benchmark each constituent of the T4 memory hierarchy and find substantial overall performance improvements over its P4 predecessor. We studied how clock throttling affects compute-intensive workloads that hit power or thermal limits. Many of our findings are novel, published here for the first time. All of them can guide high-performance software developers get closer to the GPU's peak performance.

Subfields: Electrical and Electronic Engineering, Hardware and Architecture, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

In Hardware and Architecture

In ML Systems & Hardware

Citations in the corpus

The case for 4-bit precision: k-bit Inference Scaling Laws (2022, background): “For example, for modern hardware like GPUs, it usually takes more than 100 times longer to load a number than to do an arithmetic operation with that number (Jia et al., 2019; Dongarra, 2022).”
Accurate deep neural network inference using computational phase-change memory (2019, methodology): “(2) is modeled as a Gaussian distributed random variable with zero mean and standard deviation given by the fitted curve 0 2 4 6 8 10 12 89 90 91 92 93 94”
Ten Lessons From Three Generations Shaped Google’s TPUv4i : Industrial Product (2021, background): “Others found different variations in temperature [19], presumably due to running programs with different operational intensity than MLPerf.”