Personalization of End-to-End Speech Recognition on Mobile Devices for Named Entities

By Khe Chai Sim, Franccoise Beaufays, Arnaud Benard, Dhruv Guliani, A. Kabel, Nikhil Khare, Tamar Lucassen, Petr Zadrazil, Harry Zhang, Leif T. Johnson and 2 more. 2019

86 citations recorded by the source corpus.

We study the effectiveness of several techniques to personalize end-to-end speech models and improve the recognition of proper names relevant to the user. These techniques differ in the amounts of user effort required to provide supervision, and are evaluated on how they impact speech recognition performance. We propose using keyword-dependent precision and recall metrics to measure vocabulary acquisition performance. We evaluate the algorithms on a dataset that we designed to contain names of persons that are difficult to recognize. Therefore, the baseline recall rate for proper names in this dataset is very low: 2.4%. A data synthesis approach we developed brings it to 48.6%, with no need for speech input from the user. With speech input, if the user corrects only the names, the name recall rate improves to 64.4%. If the user corrects all the recognition errors, we achieve the best recall of 73.5%. To eliminate the need to upload user data and store personalized models on a server, we focus on performing the entire personalization workflow on a mobile device.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lei He
2. Sheng Zhao
3. Yanqing Liu
4. Jinyu Li
5. Xuejiao Tan
6. Jy-yong Sohn
7. Xu Tan
8. Rui Xue

In Speech & Audio

1. Lei He
2. Sheng Zhao
3. Yanqing Liu
4. Jinyu Li
5. Xuejiao Tan
6. Rui Xue
7. Linquan Liu
8. Ed Lin

Citations in the corpus

• Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “Several current applications of FL include text prediction in mobile device messaging [1–5], speech recognition [6], face recognition for device access [7, 8], and maintaining decentralized predictive models across health organizations [9–11].”
• FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, methodology): “Speech synthesis helps to reduce word error rate (WER) in customization of automatic speech recognition (ASR) especially for low resource or OOV scenarios by generating synthetic speech using text-to-speech (TTS), and using the text and speech pairs to update the ASR models or use a spelling correct”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “With the advance of TTS technologies, a new trend is to adapt E2E models with the synthesized speech generated from the new-domain text [12, 164, 185, 186].”