AI and Memory Wall

By A. Gholami, Zi-Mei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, Kurt Keutzer. 2024

374 citations recorded by the source corpus.

The availability of unprecedented unsupervised training data, along with neural scaling laws, has resulted in an unprecedented surge in model size and compute requirements for serving/training large language models. However, the main performance bottleneck is increasingly shifting to memory bandwidth. Over the past 20 years, peak server hardware floating-point operations per second have been scaling at 3.0${\times}$× per two years, outpacing the growth of dynamic random-access memory and interconnect bandwidth, which have only scaled at 1.6 and 1.4 times every two years, respectively. This disparity has made memory, rather than compute, the primary bottleneck in AI applications, particularly in serving. Here, we analyze encoder and decoder transformer models and show how memory bandwidth can become the dominant bottleneck for decoder models. We argue for a redesign in model architecture, training, and deployment strategies to overcome this memory limitation.

Subfields: Electrical and Electronic Engineering, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

1. Kurt Keutzer
2. A. Gholami
3. Michael W. Mahoney
4. Zhen Dong
5. Yakun Sophia Shao

In ML Systems & Hardware

1. A. Gholami
2. Michael W. Mahoney
3. Kurt Keutzer

Citations in the corpus

• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, background): “With the growing divergence between computational speeds and memory speeds, this problem is only going to get worse over time [12].”
• Emerging opportunities and challenges for the future of reservoir computing (2024, neutral): “Memory Taking Up by Parameters (GB)* (b) RC versus DL in terms of the number of parameters and computational cost (measured in the amount of petaflops) required for training [160].”
• SqueezeLLM: Dense-and-Sparse Quantization (2023, background): “The disparity between compute and memory bandwidth, along with the growing memory requirements of deep learning, has been termed the Memory Wall problem [20].”