Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking

By Zhe Jia, Marco Maggioni, B. Staiger, Daniele Paolo Scarpazza. 2018

371 citations recorded by the source corpus.

Every year, novel NVIDIA GPU designs are introduced. This rapid architectural and technological progression, coupled with a reluctance by manufacturers to disclose low-level details, makes it difficult for even the most proficient GPU software designers to remain up-to-date with the technological advances at a microarchitectural level. To address this dearth of public, microarchitectural-level information on the novel NVIDIA GPUs, independent researchers have resorted to microbenchmarks-based dissection and discovery. This has led to a prolific line of publications that shed light on instruction encoding, and memory hierarchy's geometry and features at each level. Namely, research that describes the performance and behavior of the Kepler, Maxwell and Pascal architectures. In this technical report, we continue this line of research by presenting the microarchitectural details of the NVIDIA Volta architecture, discovered through microbenchmarks and instruction set disassembly. Additionally, we compare quantitatively our Volta findings against its predecessors, Kepler, Maxwell and Pascal.

Subfields: ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In ML Systems & Hardware

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “As an example, the A100 GPU has 40-80GB of high bandwidth memory (HBM) with bandwidth 1.5-2.0TB/s and 192KB of on-chip SRAM per each of 108 streaming multiprocessors with bandwidth estimated around 19TB/s [46, 47].”
MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis (2019, methodology): “We believe that our model is also well-suited for hardware speciﬁc inference optimization (such as half precision on Tesla V100 (Jia et al., 2018; Dosovitskiy & Brox, 2016) and quantization (as done in Arik et al. (2017)) which will further boost inference speed.”
Accel-Sim: An Extensible Simulation Framework for Validated GPU Modeling (2018, methodology): “Numerous works have done micro-benchmarking to characterize aspects of GPU architecture for Tesla GT200 [73], Fermi [37], Maxwell [34], and recently Volta [26].”
Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUs (2019, methodology): “Non-public microarchitectural details are conﬁgured using microbenchmark results from Jia et al. [35].”
Dissecting the Graphcore IPU Architecture via Microbenchmarking (2019, background): “Even though GPUs have evolved deep memory hierarchies featuring both cache and scratchpad memories [1, 2], their fundamental approach to hiding memory latency remains the oversubscription of threads to cores and the ability to inexpensively switch among threads.”
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023, background): “0TB/s and 192KB of on-chip SRAM per each of 108 streaming multiprocessors with bandwidth estimated around 19TB/s [6, 7].”
Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications (2019, background): “GPUs have emerged as a popular choice in this context because they excel at highly parallelizable matrix operations common in DL jobs (Jia et al., 2018; Zhu et al., 2016; Jouppi et al., 2017; Abadi et al., 2016).”
Graviton: Trusted Execution Environments on GPUs (2018, methodology): “The GPU has a shared two-level TLB used while accessing the L2 cache [19].”