Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers

By Manuele Rusci, Alessandro Capotondi, Luca Benini. 2019

95 citations recorded by the source corpus.

This paper presents a novel end-to-end methodology for enabling the deployment of low-error deep networks on microcontrollers. To fit the memory and computational limitations of resource-constrained edge-devices, we exploit mixed low-bitwidth compression, featuring 8, 4 or 2-bit uniform quantization, and we model the inference graph with integer-only operations. Our approach aims at determining the minimum bit precision of every activation and weight tensor given the memory constraints of a device. This is achieved through a rule-based iterative procedure, which cuts the number of bits of the most memory-demanding layers, aiming at meeting the memory constraints. After a quantization-aware retraining step, the fake-quantized graph is converted into an inference integer-only model by inserting the Integer Channel-Normalization (ICN) layers, which introduce a negligible loss as demonstrated on INT4 MobilenetV1 models. We report the latency-accuracy evaluation of mixed-precision MobilenetV1 family networks on a STM32H7 microcontroller. Our experimental results demonstrate an end-to-end deployment of an integer-only Mobilenet network with Top1 accuracy of 68% on a device with only 2MB of FLASH memory and 512kB of RAM, improving by 8% the Top1 accuracy with respect to previously published 8 bit implementations for microcontrollers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Song Han
2. Ji Lin
3. Luke S. Zettlemoyer
4. Chuang Gan
5. Tim Dettmers
6. Yujun Lin
7. Quoc V. Le
8. Ligeng Zhu

In Computer Vision and Pattern Recognition

1. Song Han
2. Chuang Gan
3. Ji Lin
4. Luke S. Zettlemoyer
5. Luc Van Gool
6. Dacheng Tao
7. Tim Dettmers
8. Trevor Darrell

In Electrical and Electronic Engineering

1. Song Han
2. Ji Lin
3. Tim Dettmers
4. Luke S. Zettlemoyer
5. Hao Wu
6. Patrick Judd
7. Dan Alistarh
8. Elias Frantar

In Efficiency & Compression

1. Luke S. Zettlemoyer
2. Song Han
3. Tim Dettmers
4. Ji Lin
5. Chuang Gan
6. Ligeng Zhu
7. Yujun Lin
8. Dan Alistarh

In ML Systems & Hardware

1. Song Han
2. Chuang Gan
3. Luke S. Zettlemoyer
4. Hao Wu
5. Tim Dettmers
6. Ji Lin
7. Patrick Judd

Citations in the corpus

• The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “…are methods associated with blocking and grouping (Park et al., 2022; Wu et al., 2020; Jain et al., 2020; Nagel et al., 2019; Krishnamoorthi, 2018; Rusci et al., 2020), centering (Krishnamoorthi, 2018; Jacob et al., 2017), learned data types that are found through clustering (Gong et al., 2014;…”
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation (2020, background): “This is commonly referred to as “per-channel” weight quantization in literature and we follow that convention [21, 25, 26, 38, 46].”
• MCUNet: Tiny Deep Learning on IoT Devices (2020, methodology): “There is limited literature [15, 30, 40, 28] that studies machine learning on microcontrollers.”
• On-Device Training Under 256KB Memory (2022, background): “Such a small memory budget is hardly enough for the inference of deep learning models [48, 47, 7, 12, 44, 25, 45, 60], let alone the training, which requires extra computation for the backward and extra memory for intermediate activation [19].”