MCUNet: Tiny Deep Learning on IoT Devices

By Ji Lin, Wei-Ming Chen, Yujun Lin, John Cohn, Chuang Gan, Song Han. 2020

741 citations recorded by the source corpus.

Machine learning on tiny IoT devices based on microcontroller units (MCU) is appealing but challenging: the memory of microcontrollers is 2-3 orders of magnitude less even than mobile phones. We propose MCUNet, a framework that jointly designs the efficient neural architecture (TinyNAS) and the lightweight inference engine (TinyEngine), enabling ImageNet-scale inference on microcontrollers. TinyNAS adopts a two-stage neural architecture search approach that first optimizes the search space to fit the resource constraints, then specializes the network architecture in the optimized search space. TinyNAS can automatically handle diverse constraints (i.e. device, latency, energy, memory) under low search costs. TinyNAS is co-designed with TinyEngine, a memory-efficient inference library to expand the design space and fit a larger model. TinyEngine adapts the memory scheduling according to the overall network topology rather than layer-wise optimization, reducing the memory usage by 2.7x, and accelerating the inference by 1.7-3.3x compared to TF-Lite Micro and CMSIS-NN. MCUNet is the first to achieves >70% ImageNet top1 accuracy on an off-the-shelf commercial microcontroller, using 3.6x less SRAM and 6.6x less Flash compared to quantized MobileNetV2 and ResNet-18. On visual&audio wake words tasks, MCUNet achieves state-of-the-art accuracy and runs 2.4-3.4x faster than MobileNetV2 and ProxylessNAS-based solutions with 2.2-2.6x smaller peak SRAM. Our study suggests that the era of always-on tiny machine learning on IoT devices has arrived.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Song Han
2. Chuang Gan
3. Ji Lin
4. Ligeng Zhu
5. Hongpeng Liao
6. Chien-Yao Wang
7. Haotian Tang
8. Alexey Bochkovskiy

In Computer Vision and Pattern Recognition

1. Song Han
2. Chuang Gan
3. Hongpeng Liao
4. Chien-Yao Wang
5. Ji Lin
6. Alexey Bochkovskiy
7. Luc Van Gool
8. Ligeng Zhu

In Efficiency & Compression

1. Chuang Gan
2. Ji Lin
3. Song Han
4. Ligeng Zhu
5. Haotian Tang
6. Wei-Ming Chen
7. Jiaming Tang
8. Shang Yang

In ML Systems & Hardware

1. Song Han
2. Chuang Gan
3. Ji Lin

Citations in the corpus

• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, background): “It proves to be effective for various convolutional neural works (CNNs) (Han et al., 2016; Jacob et al., 2018; Nagel et al., 2019; Wang et al., 2019; Lin et al., 2020) and transformers (Shen et al., 2020; Kim et al., 2021; Liu et al., 2021; Wang et al., 2020; Bondarenko et al., 2021).”
• Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “MCUNet [264] is a framework incorporating compact model design and compiler optimization to eliminate instruction and memory redundancy.”
• AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration (2023, positive): “Quantization reduces the bit-precision of deep learning models (Han et al., 2016; Jacob et al., 2018; Nagel et al., 2019; Wang et al., 2019; Nagel et al., 2020; Lin et al., 2020), which helps to reduce the model size and accelerate inference.”
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023, background): “Quantization reduces the bit-precision of deep learning models [21, 24, 35, 49, 34, 30], which helps to reduce the model size and accelerate inference.”
• YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors (2022, background): “For example, the devel-Figure opment of MCUNet [49, 48] and NanoDet [54] focused on producing low-power single-chip and improving the inference speed on edge CPU.”
• Cellular, Wide-Area, and Non-Terrestrial IoT: A Survey on 5G Advances and the Road Toward 6G (2021, background): “The memory (static random-access memory (SRAM)) of microcontrollers is about 3 orders of magnitude smaller than that of mobile phones and 5-6 orders of magnitude smaller than cloud GPUs [362], making it impossible to run off-the-shelf DNN models, even the compact models like MobileNetV2 [360].”
• A Review on Deep Learning in UAV Remote Sensing (2021, methodology): “MCUNet [Lin et al., 2020] combines TinyNAS and TinyEngine to build a model that requires 320kB of memory and 1MB of storage.”
• On-Device Training Under 256KB Memory (2022, methodology): “We train on VWW for 10 epochs following [48].”