AutoTM: Automatic Tensor Movement in Heterogeneous Memory Systems using Integer Linear Programming

By Mark Hildebrand, Jawad Ali Khan, Sanjeev N. Trika, Jason Lowe-Power, Venkatesh Akella. 2020

91 citations recorded by the source corpus.

Memory capacity is a key bottleneck for training large scale neural networks. Intel® Optane#8482; DC PMM (persistent memory modules) which are available as NVDIMMs are a disruptive technology that promises significantly higher read bandwidth than traditional SSDs at a lower cost per bit than traditional DRAM. In this work we show how to take advantage of this new memory technology to minimize the amount of DRAM required without compromising performance significantly. Specifically, we take advantage of the static nature of the underlying computational graphs in deep neural network applications to develop a profile guided optimization based on Integer Linear Programming (ILP) called AutoTM to optimally assign and move live tensors to either DRAM or NVDIMMs. Our approach can replace 50% to 80% of a system's DRAM with PMM while only losing a geometric mean 27.7% performance. This is a significant improvement over first-touch NUMA, which loses 71.9% of performance. The proposed ILP based synchronous scheduling technique also provides 2x performance over using DRAM as a hardware-controlled cache for very large networks.

Subfields: Model Architectures, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

ZeRO-Offload: Democratizing Billion-Scale Model Training (2021, methodology): “The third approach uses an external memory such as the CPU memory as an extension of GPU memory to increase memory capacity during training [8, 9, 11, 17, 23, 24, 33].”
OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks (2022, background): “This has been demonstrated to effectively offload the data stored on a GPU device onto the host memory [64, 38, 54], but again increases training time due to extra memory transfers.”
Sentinel: Efficient Tensor Migration and Allocation on Heterogeneous Memory Systems for Deep Learning (2021, methodology): “Most of them focus on mitigating GPU-side memory space limitation by leveraging larger CPU-side system memory [5], [6], [8]–[11], while a recent study demonstrates a HM that uses a persistent memory to scale the CPU-side DRAM capacity [7].”