TVM: An Automated End-to-End Optimizing Compiler for Deep Learning

By Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Q. Yan, Haichen Shen, M. Cowan, Yuwei Hu, Luis Ceze, Carlos Guestrin and 1 more. 2018

357 citations recorded by the source corpus.

There is an increasing need to bring machine learning to a wide diversity of hardware devices. Current frameworks rely on vendor-specific operator libraries and optimize for a narrow range of server-class GPUs. Deploying workloads to new platforms -- such as mobile phones, embedded devices, and accelerators (e.g., FPGAs, ASICs) -- requires significant manual effort. We propose TVM, a compiler that exposes graph-level and operator-level optimizations to provide performance portability to deep learning workloads across diverse hardware back-ends. TVM solves optimization challenges specific to deep learning, such as high-level operator fusion, mapping to arbitrary hardware primitives, and memory latency hiding. It also automates optimization of low-level programs to hardware characteristics by employing a novel, learning-based cost modeling method for rapid exploration of code optimizations. Experimental results show that TVM delivers performance across hardware back-ends that are competitive with state-of-the-art, hand-tuned libraries for low-power CPU, mobile GPU, and server-class GPUs. We also demonstrate TVM's ability to target new accelerator back-ends, such as the FPGA-based generic deep learning accelerator. The system is open sourced and in production use inside several major companies.

Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Hardware and Architecture, Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Efficiency & Compression

In ML Systems & Hardware

Citations in the corpus

HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “, 2018) and TVM (Chen et al., 2018).”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “, 2018) and TVM (Chen et al., 2018).”
Supporting Very Large Models using Automatic Dataflow Graph Partitioning (2018, background): “A more promising direction is to leverage recent operator code-generation tools such as TVM [30] and TC [31].”
Optimizing data-intensive computations in existing libraries with split annotations (2018, background): “Many of these systems [7, 34, 36, 46, 56] require reimplementing each operator in entirety to obtain any benefits.”
TurboTransformers: an efficient GPU serving system for transformer models (2020, methodology): “ire extra memory space for intermediate tensors and eliminate the opportunity for operator fusion. Second, currently existing DL inference frameworks such as onnxruntime [18], TenorFlow XLA [10], TVM [3], tensorRT [21] use techniques 1 arXiv:2010.05680v1 [cs.DC] 9 Oct 2020 A preprint version, 2020 J”
Deep Learning in Mobile and Wireless Networking: A Survey (2018, background): “Beyond these works, researchers also successfully adapt deep learning architectures through other designs and sophisticated optimizations, such as parameters quantization [524], [529], model slimming [536], sparsification and separation [525], representation and memory sharing [98], [530], convoluti”
A TinyML Platform for On-Device Continual Learning With Quantized Latent Replays (2021, methodology): “Examples of this trend include non-vendor-locked tools such as Google TFLite Micro [6], ARM CMSIS-NN [37], Apache TVM [38], as well as frameworks that only support specific families of devices, such as STMicroelectronics X-CUBE-AI4, GreenWaves Technologies NNTOOL5, and DORY [39].”
Understanding Reuse, Performance, and Hardware Cost of DNN Dataflow: A Data-Centric Approach (2018, background): “We validated MAESTRO’s performance model against RTL simulations of two accelerators - MAERI [24] and Eyeriss [13] when running VGG16 and AlexNet respectively 1 .”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “TVM leverage the efficiency of quantization by enabling deployment of quantized models from PyTorch and TF-Lite.”
Orca: A Distributed Serving System for Transformer-Based Generative Models (2022, background): “These systems can be seen as an abstraction sitting atop underlying model execution engines such as TensorRT [6], TVM [14], TensorFlow [8], and many others [44, 46], being agnostic to various kinds of ML models, execution engines, and computing hardware.”
The Deep Learning Compiler: A Comprehensive Survey (2020, methodology): “Rapidly, several popular DL compilers have been proposed such as TVM [25],”
Learning to Optimize Tensor Programs (2018, methodology): “We use primitives from an existing code generation framework [9] to form Se.”
MCUNet: Tiny Deep Learning on IoT Devices (2020, methodology): “Existing frameworks such as TensorFlow Lite Micro [3], CMSIS-NN [28], CMix-NN [8], and MicroTVM [9] have several limitations: 1.”