Superneurons: dynamic GPU memory management for training deep neural networks

By Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, Tim Kraska. 2018

294 citations recorded by the source corpus.

Going deeper and wider in neural architectures improves their accuracy, while the limited GPU DRAM places an undesired restriction on the network design domain. Deep Learning (DL) practitioners either need to change to less desired network architectures, or nontrivially dissect a network across multiGPUs. These distract DL practitioners from concentrating on their original machine learning tasks. We present SuperNeurons: a dynamic GPU memory scheduling runtime to enable the network training far beyond the GPU DRAM capacity. SuperNeurons features 3 memory optimizations, Liveness Analysis, Unified Tensor Pool, and Cost-Aware Recomputation; together they effectively reduce the network-wide peak memory usage down to the maximal memory usage among layers. We also address the performance issues in these memory-saving techniques. Given the limited GPU DRAM, SuperNeurons not only provisions the necessary memory for the training, but also dynamically allocates the memory for convolution workspaces to achieve the high performance. Evaluations against Caffe, Torch, MXNet and TensorFlow have demonstrated that SuperNeurons trains at least 3.2432 deeper network than current ones with the leading performance. Particularly, SuperNeurons can train ResNet2500 that has 104 basic network layers on a 12GB K40c.

Subfields: Statistical and Nonlinear Physics, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical and Nonlinear Physics

In Efficiency & Compression

Citations in the corpus

ZeRO: Memory optimizations Toward Training Trillion Parameter Models (2019, background): “1) Reducing Activation Memory: Multiple efforts have focused on reducing the memory footprint of activations through compression [15], activation checkpointing [8], [16], or live analysis [17].”
ZeRO-Offload: Democratizing Billion-Scale Model Training (2021, methodology): “The third approach uses an external memory such as the CPU memory as an extension of GPU memory to increase memory capacity during training [8, 9, 11, 17, 23, 24, 33].”
Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, methodology): “Swapping [23, 42, 55], recomputation [7, 24] and their combination [40] have been utilized to reduce the peak memory of training.”
SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping (2020, background): “However, A promising approach to address the GPU memory limitation without affecting accuracy is to swap tensor data be-tween GPU and CPU memory during DNN computation [26, 30, 40, 49].”
Capuchin: Tensor-based GPU Memory Management for Deep Learning (2020, methodology): “In prior works [22, 24, 31], each time a tensor has been swapped out, the next computation can only start when current computation that uses this tensor and this tensor’s swapping-out both fnish.”
A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication (2018, background): “In particular, distributed stochastic gradient descent has been extensively studied in both deep learning and high-performance computing communities, with the goal of accelerating large-scale learning tasks [13, 10, 9, 47, 3, 14, 30, 31, 6, 44, 38].”
Efficient Memory Management for GPU-based Deep Learning Systems (2019, methodology): “Relevant works are seen in MXNet-memonger [31], SuperNeurons [33], and the DenseNets implementation via PyTorch [34], all of which have the cost-awareness idea to selectively drop the feature maps which are easy to recompute.”
FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models (2022, background): “SuperNeurons [35] places large models on a single GPU by a fine-grained memory management method.”
Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads (2019, background): “Therefore, vDNN [39] proposes virtualized memory manager, and SuperNeurons [49] adopts ﬁne-grained layer-wise memory control to schedule memory ﬂexibly between CPU and GPU.”
Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUs (2019, background): “Many domain-specific solutions have been proposed across the stack to address the DL memory capacity challenge, including DL-specific footprint reduction schemes [45], [46], [25] and asynchronous offloading of data during each training iteration [47], [14], [12], [48], [15], [49], [50].”
AutoTM: Automatic Tensor Movement in Heterogeneous Memory Systems using Integer Linear Programming (2020, methodology): “SuperNeurons [44] and moDNN [8] build on vDNN. SuperNeurons introduces a run-time manager for offloading and prefetching tensors between GPU and CPU memory as well as a cost-aware method of applying recomputation of forward pass layers during the backward pass to reduce memory.”
Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect (2019, methodology): “As part of the community effort, we are planning to pursue these research directions in our future work with our past experience on GPU analytic modeling [38], [39], [40] and performance optimization [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51].”
Zico: Efficient GPU Memory Sharing for Concurrent DNN Training (2021, methodology): “Other prior works study dropping tensors created in forward pass and recomputing them in backward pass [6, 19, 41].”
An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning (2019, unknown): “Although deep learning has successfully been applied to solving computationally intensive learning tasks in many domains [15, 17, 18, 24, 48], there are few studies that have used deep learning techniques to solve database tuning problems[49].”
Wireless Edge Computing With Latency and Reliability Guarantees (2019, background): “3 GB [98]) cannot fit into a single device’s memory and has to be partitioned into multiple segments stored across edge devices and server, i.”
vPipe: A Virtualized Acceleration System for Achieving Efficient and Scalable Pipeline Parallel DNN Training (2022, methodology): “SuperNeuron [56] and Capuchin [37] coherently combine swap and recompute to dynamically reduce the memory consumption of DNN training on a single GPU.”
Sentinel: Efficient Tensor Migration and Allocation on Heterogeneous Memory Systems for Deep Learning (2021, methodology): “Lack of memory causes DNN training to have out-of-memory crashes and limits the sizes of the model and batch for training, causing degradation in training effectiveness and efficiency [5], [6].”
Multi-tenant GPU Clusters for Deep Learning Workloads: Analysis and Implications (2018, background): “Therefore, vDNN [39] proposes virtualized memory manager, and SuperNeurons [51] adopts fine-grained layer-wise memory control to schedule memory flexibly between CPU and GPU.”