The tensor algebra compiler
By Fredrik Kjolstad, Shoaib Kamil, Stephen Chou, D. Lugato, Saman P. Amarasinghe. Proc. ACM Program. Lang., 2017
456 citations recorded by the source corpus.
Tensor algebra is a powerful tool with applications in machine learning, data analytics, engineering and the physical sciences. Tensors are often sparse and compound operations must frequently be computed in a single kernel for performance and to save memory. Programmers are left to write kernels for every operation of interest, with different mixes of dense and sparse tensors in different formats. The combinations are infinite, which makes it impossible to manually implement and optimize them all. This paper introduces the first compiler technique to automatically generate kernels for any compound tensor algebra operation on dense and sparse tensors. The technique is implemented in a C++ library called taco. Its performance is competitive with best-in-class hand-optimized kernels in popular libraries, while supporting far more tensor operations.
Subfields: Artificial Intelligence, Computational Mathematics, Hardware and Architecture
View Paper: https://www.semanticscholar.org/paper/c2e1139691c3a337831e36ee7afeab8817ab5d48
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Mohit Bansal
- Yoshua Bengio
- Karen Simonyan
- Jianfeng Gao
- Abhinav Kumar Gupta
- Yejin Choi
- Tao Lei
In Computational Mathematics
In Hardware and Architecture
- Albert Cohen
- Nicolas Vasilache
- Oleksandr Zinenko
- Andrew Adams
- William S. Moses
- Theodoros Theodoridis
- Zach DeVito
- Priya Goyal
Citations in the corpus
- Learning to Optimize Tensor Programs (2018, background): “There are many domain-specific languages (DSLs) for code generation [32, 36, 15, 37, 20, 30], each with with a different E , Se and g.”
- Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions (2018, methodology): “OoLaLa [53] takes a similar approach for linear algebra, and TACO [46] and Simit [47] use a notation similar to that of TC, but generate sparse matrix code for numerical solvers.”
- Fast Sparse ConvNets (2019, methodology): “For a sparse baseline we use the kernel generated by the TACO compiler [23].”
- Timeloop: A Systematic Approach to DNN Accelerator Evaluation (2019, background): “Other future work includes modeling accelerators for other domains like graph analytics [40] and sparse tensor algebra [20].”
- TVM: An Automated End-to-End Optimizing Compiler for Deep Learning (2018, methodology): “Our tensor expression takes cues from Halide [32], Darkroom [17], and TACO [23].”
- Structured Pruning of Large Language Models (2019, methodology): “For models obtained using unstructured AGP, we experimented with the sparse matrix multiplication rou-Figure tine provided in Pytorch (Paszke et al., 2017) and a recent linear algebra compiler (Kjolstad et al., 2017), but were unable to achieve a speedup.”
- TVM: An Automated End-to-End Optimizing Compiler for Deep Learning (2018, background): “TACO [22] introduces a generic way to generate sparse tensor operators on CPU.”