The Deep Learning Compiler: A Comprehensive Survey

By Mingzhen Li, Yi Liu, Xiaoyan Liu, Qingxiao Sun, Xin You, Hailong Yang, Zhongzhi Luan, Depei Qian. 2020

248 citations recorded by the source corpus.

The difficulty of deploying various deep learning (DL) models on diverse DL hardware has boosted the research and development of DL compilers in the community. Several DL compilers have been proposed from both industry and academia such as Tensorflow XLA and TVM. Similarly, the DL compilers take the DL models described in different DL frameworks as input, and then generate optimized codes for diverse DL hardware as output. However, none of the existing survey has analyzed the unique design architecture of the DL compilers comprehensively. In this article, we perform a comprehensive survey of existing DL compilers by dissecting the commonly adopted design in details, with emphasis on the DL oriented multi-level IRs, and frontend/backend optimizations. We present detailed analysis on the design of multi-level IRs and illustrate the commonly adopted optimization techniques. Finally, several insights are highlighted as the potential research directions of DL compiler. This is the first survey article focusing on the design architecture of DL compilers, which we hope can pave the road for future research towards DL compiler.

Subfields: Artificial Intelligence, Hardware and Architecture, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Christopher R'e
2. Tri Dao
3. Atri Rudra
4. Stefano Ermon
5. Daniel Y. Fu
6. Tailin Liang
7. M. G. Sarwar Murshed
8. C. John Glossner

In Hardware and Architecture

1. Tri Dao
2. Christopher R'e
3. Atri Rudra
4. Stefano Ermon
5. Daniel Y. Fu
6. Ganesh Ananthanarayanan

In Optimization & Training

1. Tri Dao
2. Christopher R'e
3. Atri Rudra
4. Stefano Ermon
5. Tailin Liang
6. C. John Glossner
7. Bashir I. Morshed
8. Shaobo Shi

Citations in the corpus

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “[51] Mingzhen Li, Yi Liu, Xiaoyan Liu, Qingxiao Sun, Xin You, Hailong Yang, Zhongzhi Luan, Lin Gan, Guangwen Yang, and Depei Qian.”
• Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, background): “Different frameworks and compilers for ML are surveyed in [13] having no consideration of edge technology and compression-compiler codesign.”
• Machine Learning at the Network Edge: A Survey (2019, background): “Li et al. provide a detailed survey of the architectures of deep learning compilers [82].”
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “A detailed survey can be found in [140].”