SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping

By Chien-chin Huang, Gu Jin, Guozhang Jin, Jinyang Li. 2020

233 citations recorded by the source corpus.

It is known that deeper and wider neural networks can achieve better accuracy. But it is difficult to continue the trend to increase model size due to limited GPU memory. One promising solution is to support swapping between GPU and CPU memory. However, existing work on swapping only handle certain models and do not achieve satisfactory performance. Deep learning computation is commonly expressed as a dataflow graph which can be analyzed to improve swapping. We propose SwapAdvisor, which performs joint optimization along 3 dimensions based on a given dataflow graph: operator scheduling, memory allocation, and swap decisions. SwapAdvisor explores the vast search space using a custom-designed genetic algorithm. Evaluations using a variety of large models show that SwapAdvisor can train models up to 12 times the GPU memory limit while achieving 53-99% of the throughput of a hypothetical baseline with infinite GPU memory.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

ZeRO-Offload: Democratizing Billion-Scale Model Training (2021, methodology): “The third approach uses an external memory such as the CPU memory as an extension of GPU memory to increase memory capacity during training [8, 9, 11, 17, 23, 24, 33].”
Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, methodology): “Swapping [23, 42, 55], recomputation [7, 24] and their combination [40] have been utilized to reduce the peak memory of training.”
Pre-Trained Models: Past, Present and Future (2021, background): “Additionally, some researchers found that they could get even better performance on benchmarks when training one model with several languages comparing with training several monolingual models (Lample and Conneau, 2019; Huang et al., 2020b).”
ASPEN: High-Throughput LoRA Fine-Tuning of Large Language Models with a Single GPU (2023, unknown): “These pre-trained models can be fine-tuned for various downstream jobs – a method that has been proven effective [21].”
Sentinel: Efficient Tensor Migration and Allocation on Heterogeneous Memory Systems for Deep Learning (2021, methodology): “Because of the GA algorithm SwapAdvisor uses to decide tensor migration, SwapAdvisor suggests to use 30 minutes to make the final migration decision at runtime [8]; According to our experimental results, for a large model such as BERT-large, SwapAdvisor cannot make the decision within 30 minutes.”
This paper is included in the Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation. (unknown): “In addition to parallelization, there are other complementary techniques for training large-scale models, such as memory optimization [12, 14,21,23,28,46], communication compression [6,53], and low-precision training [35].”
High-throughput Generative Inference of Large Language Models with a Single GPU (2023, background): “Within broader domains, memory optimizations and offloading have been studied for training (Huang et al., 2020; Ren et al., 2021; Steiner et al., 2022) and linear algebra (Jia-Wei & Kung, 1981; Demmel, 2013).”
Training Neural Networks on Tiny Devices with Integrated Rematerialization and Paging (2022, background): “Paging: Huang et al. (2020) and Ren et al. (2021) page activations off a memory-scarce GPU to the CPU when out of memory.”
PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications (2020, background): “Other works like vDNN [43] and SwapAdvisor [44] also have GPU memory management module, but they focus on memory management for a single training task of large models, which are not directly comparable to PipeSwitch.”
Zico: Efficient GPU Memory Sharing for Concurrent DNN Training (2021, background): “host memory as a swap storage for DNN training to mitigate memory footprint in GPU [17,32,36].”
Wavelet: Efficient DNN Training with Tick-Tock Scheduling (2021, background): “…can increase GPU utilization, it introduces extra overheads, such as frequent context switching and data loading from storage (Mohan et al., 2021), maintaining multiple models within the same GPU memory (Yu & Chowdhury, 2020; Huang et al., 2020), inter-job interference (Xiao et al., 2018), etc.”