Triton: an intermediate language and compiler for tiled neural network computations

By Philippe Tillet, Hsiang-Tsung Kung, David D. Cox. 2019

805 citations recorded by the source corpus.

The validation and deployment of novel research ideas in the field of Deep Learning is often limited by the availability of efficient compute kernels for certain basic primitives. In particular, operations that cannot leverage existing vendor libraries (e.g., cuBLAS, cuDNN) are at risk of facing poor device utilization unless custom implementations are written by experts – usually at the expense of portability. For this reason, the development of new programming abstractions for specifying custom Deep Learning workloads at a minimal performance cost has become crucial. We present Triton, a language and compiler centered around the concept of tile, i.e., statically shaped multi-dimensional sub-arrays. Our approach revolves around (1) a C-based language and an LLVM-based intermediate representation (IR) for expressing tensor programs in terms of operations on parametric tile variables and (2) a set of novel tile-level optimization passes for compiling these programs into efficient GPU code. We demonstrate how Triton can be used to build portable implementations of matrix multiplication and convolution kernels on par with hand-tuned vendor libraries (cuBLAS / cuDNN), or for efficiently implementing recent research ideas such as shift convolutions.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Hardware and Architecture, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Hardware and Architecture

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, methodology): “However, we also publish a version of efficient implementation in Triton [37].”
Stable and low-precision training for large-scale vision-language models (2023, methodology): “We provide open-source Triton [57] kernels for Switchback to enable future work on efficient quantization schemes.”
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (2025, neutral): “Since the inner-loop length (proportional to the selected block count n ) remains nearly identical for different query blocks, we put query/output loops in Triton’s grid scheduler to simplify and optimize the kernel.”
AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration (2023, positive): “Our concurrent work, MLC-LLM (MLC-Team, 2023) offers strong results on multiple edge CPU and GPU platforms thanks to the powerful TVM (Chen et al., 2018; Feng et al., 2023) backend.”
Learning to (Learn at Test Time): RNNs with Expressive Hidden States (2024, neutral): “Next, we discuss two relevant lines of work in detail: test-time training and fast weights.”
LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, background): “It is worth noting that our system is currently implemented in Triton (Tillet et al., 2019), and further porting it to C++ could yield even greater speedup.”
Ansor : Generating High-Performance Tensor Programs for Deep Learning (2020, background): “It cannot outperform TVM on operators like conv2d and matmul [11,48].”
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023, methodology): “We also outperform a recent Triton [46] implementation [40] for GPTQ by 2.”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (2023, methodology): “This fused implementation (in Triton (Tillet et al., 2019)) yields up to 4 × speedup compared to a standard PyTorch implementation (Appendix E).”
Gated Linear Attention Transformers with Hardware-Efficient Training (2023, methodology): “For training speed, the GLA Transformer has significantly higher throughput than a similarly sized Mamba model.”
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2024, neutral): “Additionally, when using a higher sparsity rate, the model performance may noticeably decline.”
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, neutral): “Unsloth (Han and Han, 2023) incorporates Triton (Tillet et al., 2019) for implementing the backward propagation of LoRA, which reduces floating-point operations (FLOPs) during gradient descent and leads to expedited LoRA training.”
Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, positive): “We implement both the pure recurrent form 2 and the chunkwise parallel form in Triton [109] and show the speed-ups for various sequence lengths (L) and head dimensions ( d head ) in the right ﬁgure, where the model dimension d is 2048 and we vary batch size and sequence length so that they multily…”
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers (2024, neutral): “We attach more details and benefits coming from Triton to the appendix.”
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, methodology): “In particular, we make use of the block-sparse MatMul kernel from Triton [42], and we show that block sparsity can reduce both memory footprint and inference time.”
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (2024, neutral): “× We first compare DeepSeekMoE with a larger GShard model with 1.5 times the expert size, which results in 1.”