An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models

By Khe Chai Sim, Petr Zadrazil, Franccoise Beaufays. 2019

65 citations recorded by the source corpus.

Speaker-independent speech recognition systems trained with data from many users are generally robust against speaker variability and work well for a large population of speakers. However, these systems do not always generalize well for users with very different speech characteristics. This issue can be addressed by building personalized systems that are designed to work well for each specific user. In this paper, we investigate the idea of securely training personalized end-to-end speech recognition models on mobile devices so that user data and models never leave the device and are never stored on a server. We study how the mobile training environment impacts performance by simulating on-device data consumption. We conduct experiments using data collected from speech impaired users for personalization. Our results show that personalization achieved 63.7\% relative word error rate reduction when trained in a server environment and 58.1% in a mobile environment. Moving to on-device personalization resulted in 18.7% performance degradation, in exchange for improved scalability and data privacy. To train the model on device, we split the gradient computation into two and achieved 45% memory reduction at the expense of 42% increase in training time.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Improving Federated Learning Personalization via Model Agnostic Meta Learning (2019, background): “Existing FL personalization work directly takes a converged initial model and conducts personalization evaluation via gradient descent (Beaufays et al., 2019).”
Survey of Personalization Techniques for Federated Learning (2020, background): “Most techniques for personalization [14] generally involve two discrete steps.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “There are interesting works [256, 257] which study how to do secure adaptation on devices with continuous learning so that the user data never leaves devices.”
Federated Multi-Task Learning under a Mixture of Distributions (2021, methodology): “A naive approach for FL personalization consists in learning first a global model and then fine-tuning its parameters at each client via a few iterations of stochastic gradient descent [58].”
Federated Learning for Healthcare Informatics (2019, background): “Although there are some general work about federated learning personalization [48, 94], for healthcare informatics, how to combining the medical domain knowledge and make the global model be personalized for every medical institutions or wearable devices is another open question.”