Machine Learning Systems are Stuck in a Rut
By Paul Barham, Michael Isard. 2019
78 citations recorded by the source corpus.
In this paper we argue that systems for numerical computing are stuck in a local basin of performance and programmability. Systems researchers are doing an excellent job improving the performance of 5-year-old benchmarks, but gradually making it harder to explore innovative machine learning research ideas. We explain how the evolution of hardware accelerators favors compiler back ends that hyper-optimize large monolithic kernels, show how this reliance on high-performance but inflexible kernels reinforces the dominant style of programming model, and argue these programming abstractions lack expressiveness, maintainability, and modularity; all of which hinders research progress. We conclude by noting promising directions in the field, and advocate steps to advance progress towards high-performance general purpose numerical computing systems on modern accelerators.
Subfields: Artificial Intelligence, ML Systems & Hardware
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Yi Tay
- Mostafa Dehghani
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Ruslan Salakhutdinov
- Yonghui Wu
In ML Systems & Hardware
- Geoffrey E. Hinton
- Anurag Arnab
- Mostafa Dehghani
- Yi Tay
- Koushik Sen
- Sanjay Ghemawat
- Michael Isard
- Lianmin Zheng
Citations in the corpus
- Pathways: Asynchronous Distributed Dataflow for ML (2022, background): “Models like Mixture of Experts (MoE) (Shazeer et al., 2017) and routed capsule networks (Hinton et al., 2018; Barham and Isard, 2019) exploit computational sparsity by “routing” different (sub-)examples to the accelerators hosting different subsets of model weights based on learned functions that…”
- The Efficiency Misnomer (2021, background): “In (Barham and Isard, 2019), the authors make an excellent case for improving the programmability of software stack for modern accelerators to enable a wider class of models.”
- How to Represent Part-Whole Hierarchies in a Neural Network (2021, methodology): “With considerable effort, models that use capsules have achieved some suc-7 cesses in supervised and unsupervised learning on small datasets [Sabour et al., 2017, Hinton et al., 2018, Kosiorek et al., 2019], but they have not scaled well to larger datasets [Barham and Isard, 2019].”
- The hardware lottery (2020, neutral): “(Barham & Isard, 2019).”
- Ansor : Generating High-Performance Tensor Programs for Deep Learning (2020, background): “The significant manual effort required to produce efficient operator implementations for each target accelerator limits the development and innovation of new operators [7] and specialized accelerators [35].”