vDNN: Virtualized deep neural networks for scalable, memory-efficient neural network design

By Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, A. Zulfiqar, S. Keckler. 2016

477 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Efficiency & Compression

Citations in the corpus

ZeRO: Memory optimizations Toward Training Trillion Parameter Models (2019, background): “2) CPU Offload: [18], [19] exploit heterogeneous nature of today’s compute nodes, offloading model states to CPU memory through algorithmic design, virtualized memory, respectively.”
Pathways: Asynchronous Distributed Dataflow for ML (2022, methodology): “Large models (Brown et al., 2020) can be limited by available accelerator memory, requiring GPU memory virtualization (Rhu et al., 2016; Ausavarungnirun et al., 2018) or DRAM ofﬂoad (Rajbhandari et al., 2021).”
Supporting Very Large Models using Automatic Dataflow Graph Partitioning (2018, methodology): “Swapping [4, 35, 36] is a baseline that swaps in/out memory blocks with nearly-inﬁnite CPU memory.”
Superneurons: dynamic GPU memory management for training deep neural networks (2018, methodology): “Under Data Parallelism, vDNN [19] proposes a prefetching and offloading technique to utilize the CPU DRAM as an external buffer for the GPU.”
SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping (2020, background): “However, A promising approach to address the GPU memory limitation without affecting accuracy is to swap tensor data be-tween GPU and CPU memory during DNN computation [26, 30, 40, 49].”
SCNN: An accelerator for compressed-sparse convolutional neural networks (2017, background): “Nonetheless, the computation requirements for inference can be enormous, particularly with the emergence of deeper networks (hundreds of layers [19, 20, 29]) and larger inputs sets, such as high-definition video.”
Deep Convolutional Neural Network for Inverse Problems in Imaging (2016, background): “Usually, the memory complexity becomes O(N2 RL) in order to include all the feature maps in the process [59].”
Gist: Efficient Data Encoding for Deep Neural Network Training (2018, methodology): “Generic Approaches. vDNN transfers the data between CPU and GPU memory using smart prefetching analysis [39], ﬁtting large networks in the GPU memory, but at the expense of (1) performance cost (15% on average, and upto 27%), and (2) energy cost of using PCIe and GPU DRAM bus constantly for the…”
Sentinel: Efficient Tensor Migration and Allocation on Heterogeneous Memory Systems for Deep Learning (2021, methodology): “On the GPU-based system, Sentinel enables larger batch size in training by 1.9X and higher training throughput by 2X than vDNN [6], and enables comparable batch size and higher training throughput by 16%, 17% and 65% than three state-of-the-art solutions (Capuchin [9], AutoTM [7] and SwapAdvisor…”
The Architectural Implications of Autonomous Driving: Constraints and Acceleration (2018, background): “In our real-system characterization, we identify three computational bottlenecks, which conventional multicore CPUs are incapable of processing under the identiied design constraints.”
Capuchin: Tensor-based GPU Memory Management for Deep Learning (2020, methodology): “In graph mode, the second baseline is vDNN [24], which ofoads the appropriate memory and uses a static prefetching strategy to overlap data transfer and computation time.”
Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads (2019, methodology): “While there are software mechanisms to enable sharing, they often have high overhead making it challenging to share resources across jobs [40, 53].”
Parallel Training of Pre-Trained Models via Chunk-Based Dynamic Memory Management (2021, unknown): “Activation offloading [1, 2, 36, 39] is an alternative approach that consists in offloading some of forward activations from GPU to CPU.”
AutoTM: Automatic Tensor Movement in Heterogeneous Memory Systems using Integer Linear Programming (2020, background): “We did not compare our results directly against vDNN [36] for two reasons.”