Personalized speech recognition on mobile devices

By Ian McGraw, Rohit Prabhavalkar, Raziel Alvarez, Montse Gonzalez Arenas, Kanishka Rao, David Rybach, Ouais Alsharif, Hasim Sak, Alexander Gruenstein, Franccoise Beaufays and 1 more. 2016

201 citations recorded by the source corpus.

We describe a large vocabulary speech recognition system that is accurate, has low latency, and yet has a small enough memory and computational footprint to run faster than real-time on a Nexus 5 Android smartphone. We employ a quantized Long Short-Term Memory (LSTM) acoustic model trained with connectionist temporal classification (CTC) to directly predict phoneme targets, and further reduce its memory footprint using an SVD-based compression scheme. Additionally, we minimize our memory footprint by using a single language model for both dictation and voice command domains, constructed using Bayesian interpolation. Finally, in order to properly handle device-specific information, such as proper names and other context-dependent information, we inject vocabulary items into the decoder graph and bias the language model on-the-fly. Our system achieves 13.5% word error rate on an open-ended dictation task, running with a median speed that is seven times faster than real-time.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Tara N. Sainath
3. Rohit Prabhavalkar
4. Dong Yu
5. Yonghui Wu
6. Yu Zhang
7. Bo Li
8. Chung-Cheng Chiu

In Signal Processing

1. Jinyu Li
2. Tara N. Sainath
3. Rohit Prabhavalkar
4. Dong Yu
5. Yonghui Wu
6. Bo Li
7. Chung-Cheng Chiu
8. Yu Zhang

Citations in the corpus

• Personalization of End-to-End Speech Recognition on Mobile Devices for Named Entities (2019, methodology): “This includes techniques such as biasing [7, 8, 9], shallow fusion [10], deep fusion [10] and cold fusion [11].”
• Streaming End-to-end Speech Recognition for Mobile Devices (2018, methodology): “In order to reduce memory consumption, both on disk and at runtime, and to optimize the model’s execution to meet real-time requirements, we quantize parameters from 32-bit floating-point precision into 8-bit fixed-point, as in our previous work [9].”
• Federated User Representation Learning (2019, background): “Collaborative personalization, like learning user embeddings jointly with the task, is a powerful way to improve accuracy of neural-network-based models by adapting the model to each user’s behavior [8, 23, 14, 11, 21, 31].”
• On the compression of recurrent neural networks with an application to LVCSR acoustic modeling for embedded speech recognition (2016, methodology): “3 MB, thus enabling them to be run many times faster than real-time on recent mobile devices [28].”
• An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models (2019, methodology): “g using contact information from on-device address books) [25].”
• Gpu-Accelerated Viterbi Exact Lattice Decoder for Batched Online and Offline Speech Recognition (2019, background): “Tight latency requirements guided work to improve inference performance of models deployed in datacenters and encouraged research on supporting inference at the edge, including low-power devices [2, 3].”
• Mobile Keyboard Input Decoding with Finite-State Transducers (2017, methodology): “This places keyboard somewhere in between our server-based speech recognizer which relies on triphone models and our embedded recognizer which uses monophones [8].”
• Streaming small-footprint keyword spotting using sequence-to-sequence models (2017, methodology): “In the second, we apply an n-gram phone LM, trained on automatically generated phonetic transcriptions of the same utterances that are used to train the word-level LM in [33]; the number of parameters in the phone LM is trained to match the number of parameters of the word LM in [33].”
• On Online Attention-Based Speech Recognition and Joint Mandarin Character-Pinyin Training (2016, background): “This limits the deployments of hybrid ASR systems without compacting the LM [8].”
• Knowledge Distillation for Sequence Model (2018, result): “With 10 times more parameters, the teacher model obtains 20% reduction in WER versus that of the student model, which is similar to the observation in [23].”
• Deep Learning in Mobile and Wireless Networking: A Survey (2018, background): “[248] Personalised speech recognition Edge-based LSTM Prabhavalkar et al.”
• Large-Scale Visual Speech Recognition (2018, methodology): “Such techniques are extensively used in speech recognition (e.g. Miao et al., 2015; McGraw et al., 2016); we refer the reader to the thorough presentation of Mohri et al.”
• Two-Pass End-to-End Speech Recognition (2019, methodology): “E2E models are particularly attractive for on-device ASR, as they can outperform on-device conventional models [10] of comparable size.”
• Hello Edge: Keyword Spotting on Microcontrollers (2017, background): “Detection of keyword wakes up the device and then activates the full scale speech recognition either on device [3] or in the cloud.”