Streaming keyword spotting on mobile devices

By Oleg Rybakov, Natasha Kononenko, Niranjan A. Subrahmanya, Mirkó Visontai, Stella Laurenzo. 2020

129 citations recorded by the source corpus.

In this work we explore the latency and accuracy of keyword spotting (KWS) models in streaming and non-streaming modes on mobile phones. NN model conversion from non-streaming mode (model receives the whole input sequence and then returns the classification result) to streaming mode (model receives portion of the input sequence and classifies it incrementally) may require manual model rewriting. We address this by designing a Tensorflow/Keras based library which allows automatic conversion of non-streaming models to streaming ones with minimum effort. With this library we benchmark multiple KWS models in both streaming and non-streaming modes on mobile phones and demonstrate different tradeoffs between latency and accuracy. We also explore novel KWS models with multi-head attention which reduce the classification error over the state-of-art by 10% on Google speech commands data sets V2. The streaming library with all experiments is open-sourced.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Real-Time Speech Frequency Bandwidth Extension (2020, methodology): “This is inspired by previous work on streaming architectures for keyword spotting [2], that we extended to support the operations needed to deploy a UNet generator.”
Wav2KWS: Transfer Learning From Speech Representations for Keyword Spotting (2021, background): “Although improved end-to-end models [21] (e.g., QuartzNet [35]) for KWS are available, a pretrained model on a large-scale speech corpus has not been used for KWS. Weachieve accurate KWS with a small training dataset even if there are not additional synthesized data.”
A Streamwise Gan Vocoder for Wideband Speech Coding at Very Low Bit Rate (2021, unknown): “This condition is fulﬁlled in StyleMelGAN by adding an internal memory buffer to the causal convolutions in inference mode, as described in [16] and illustrated in Figure 2.”
AST: Audio Spectrogram Transformer (2021, methodology): “Such CNN-attention hybrid models have achieved state-of-the-art (SOTA) results for many audio classification tasks such as audio event classification [7, 8], speech command recognition [9], and emotion recognition [10].”