Benchmarking TinyML Systems: Challenges and Direction
By Colby R. Banbury, Vijay Janappa Reddi, Maximilian Lam, William Fu, A. Fazel, J. Holleman, Xinyuan Huang, R. Hurtado, David Kanter, Anton Lokhmotov and 7 more. arXiv.org, 2020
327 citations recorded by the source corpus.
Recent advancements in ultra-low-power machine learning (TinyML) hardware promises to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted benchmark for these systems. Benchmarking allows us to measure and thereby systematically compare, evaluate, and improve the performance of systems and is therefore fundamental to a field reaching maturity. In this position paper, we present the current landscape of TinyML and discuss the challenges and direction towards developing a fair and useful hardware benchmark for TinyML workloads. Furthermore, we present our three preliminary benchmarks and discuss our selection methodology. Our viewpoints reflect the collective thoughts of the TinyMLPerf working group that is comprised of 30 organizations.
Subfields: Computer Vision and Pattern Recognition, Sociology and Political Science
View Paper: https://www.semanticscholar.org/paper/9ecbd59ed85d7b6ec34d785559439c7dad09c19f
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
In Sociology and Political Science
Citations in the corpus
- On-Device Training Under 256KB Memory (2022, methodology): “We co-designed an efficient training system, Tiny Training Engine (TTE), to transform the above algorithms into slim binary codes (Figure 6).”
- A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, unknown): “A lot of edge applications are now using DNN-based algorithms [102], [103] and there is an increasing focus on making DNN inference efficient on edge devices [104], [105].”