Hello Edge: Keyword Spotting on Microcontrollers

By Yundong Zhang, Naveen Suda, Liangzhen Lai, Vikas Chandra. 2017

505 citations recorded by the source corpus.

Keyword spotting (KWS) is a critical component for enabling speech based user interactions on smart devices. It requires real-time response and high accuracy for good user experience. Recently, neural networks have become an attractive choice for KWS architecture because of their superior accuracy compared to traditional speech processing algorithms. Due to its always-on nature, KWS application has highly constrained power budget and typically runs on tiny microcontrollers with limited memory and compute capability. The design of neural network architecture for KWS must consider these constraints. In this work, we perform neural network architecture evaluation and exploration for running KWS on resource-constrained microcontrollers. We train various neural network architectures for keyword spotting published in literature to compare their accuracy and memory/compute requirements. We show that it is possible to optimize these neural network architectures to fit within the memory and compute constraints of microcontrollers without sacrificing accuracy. We further explore the depthwise separable convolutional neural network (DS-CNN) and compare it against other neural network architectures. DS-CNN achieves an accuracy of 95.4%, which is ~10% higher than the DNN model with similar number of parameters.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Federated Learning for Keyword Spotting (2018, methodology): “The resource constraints for our wake word detector are 200k parameters (based on the medium-sized model proposed in [2]), and 20 MFLOPS.”
Federated Learning with Non-IID Data (2018, methodology): “For MNIST and CIFAR-10, we use same CNN architectures as [3], where as for KWS, we use CNN architecture from [1].”
Resource-Efficient Neural Architect (2018, neutral): “Similar to [Zhang et al., 2017], we consider KWS problem with 12 classes.”
A first look into the carbon footprint of federated learning (2020, unknown): “Following the setup described in Zhang et al. (2017), we train the model to classify the audio clips into one of the 10 keywords - “Yes”, “No”, “Up”,“Down”, “Left”, “Right”, “On”, “Off”, “Stop”, “Go”, along with “silence” (i.e. no word spoken) and “unknown” word, representing the remaining 20…”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “Hello Edge [334] implements various DL models on MCU for keyword spotting for speech-based smart devices.”
Wav2KWS: Transfer Learning From Speech Representations for Keyword Spotting (2021, methodology): “While the applicability of neural networks to KWS has been demonstrated, recent studies have pursued performance improvement and reduction in the number of parameters [18]–[20], and other studies have focused on improving the real-time KWS performance [12], [21].”
Compressing RNNs for IoT devices by 15-38x using Kronecker Products (2019, methodology): “We used 2 × 2 Kronecker factor matrices for hidden-hidden/recurrent matrices of GRU layers (Cho et al., 2014) of the key-word spotting network (Zhang et al., 2017).”
Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers (2019, methodology): “Given common microcontroller architectures [25], we distinguish: • Read-Only (RO) Memory , to store frozen inference parameters, i.e. parameters that will not change during the lifetime of a smart device.”
Computational bioacoustics with deep learning: a review and roadmap (2021, methodology): “…audio, to extend deployment durations before storage is full and reduce transmission bandwidths: this is traditionally performed with simple energy detection, but could be enhanced by lightweight ML algorithms, perhaps similar to “keyword spotting” in domestic devices (Zhang et al., 2017).”
Benchmarking TinyML Systems: Challenges and Direction (2020, background): “Machine learning (ML) inference on the edge is an increasingly attractive prospect due to its potential for increasing energy efﬁciency (Fedorov et al., 2019), privacy, responsiveness (Zhang et al., 2017), and autonomy of edge devices.”
TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems (2020, background): “At one end of the spectrum, a big embedded system has a few megabytes of ﬂash ROM and at most a megabyte of SRAM.”
Efficient Keyword Spotting Using Dilated Convolutions and Gating (2018, methodology): “More advanced architectures have been successfully applied to KWS problems, such as Convolutional Neural Networks (CNNs) exploiting local dependencies [5, 6].”
MLPerf Tiny Benchmark (2021, methodology): “Model For the closed division model, we used the small depthwise-separable CNN described in [22].”
Visual Wake Words Dataset (2019, background): “Recent papers benchmarking the speed-latency tradeoffs for the keyword-spotting use-case [9] on microcontrollers propose 20–64 KB model with 10-20 million operations per second [22, 23, 24, 25].”
CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs (2018, methodology): “Depthwise separable convolution has been used to achieve compact network architectures, which are particularly useful for resource-constrained devices [12].”
Streaming keyword spotting on mobile devices (2020, methodology): “This model processes speech features by using a sequence of 2D convolutional and 2D depthwise layers followed by batch normalization with average pooling (as in [5]) and finished by applying fully connected layers.”