Federated Learning for Keyword Spotting

By David Leroy, Alice Coucke, Thibaut Lavril, Thibault Gisselbrecht, Joseph Dureau. 2018

317 citations recorded by the source corpus.

We propose a practical approach based on federated learning to solve out-of-domain issues with continuously running embedded speech-based models such as wake word detectors. We conduct an extensive empirical study of the federated averaging algorithm for the "Hey Snips" wake word based on a crowdsourced dataset that mimics a federation of wake word users. We empirically demonstrate that using an adaptive averaging strategy inspired from Adam in place of standard weighted model averaging highly reduces the number of communication rounds required to reach our target performance. The associated upstream communication costs per user are estimated at 8 MB, which is a reasonable in the context of smart home voice assistants. Additionally, the dataset used for these experiments is being open sourced with the aim of fostering further transparent research in the application of federated learning to speech data.

Subfields: Artificial Intelligence, Speech & Audio, Federated & Distributed Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Federated & Distributed Learning

In Signal Processing

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “ai is developing cross-device FL solutions for medical research [18], and Snips has explored cross-device FL for hotword detection [19].”
FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “Promising application domains include, but are not limited to, computer vision [77, 78], natural language processing [79, 80, 81, 82, 83], ﬁnance [74, 35, 84], transportation [85, 86, 87, 88, 89, 90, 91, 92, 93, 94], digital health [95, 96, 97, 98, 99, 100, 101], recommendation [102, 103, 104, 105,…”
Real-World Image Datasets for Federated Learning (2019, methodology): “Federated learning algorithms are also applied to industrial scenarios including next-word-prediction [Hard et al. , 2018], speech keyword spot-ting [Leroy et al. , 2018], and images classiﬁcation [Liu et al. , 2018].”
LEAF: A Benchmark for Federated Settings (2018, background): “…3, 11, 32, 34]; (2) realistic but proprietary federated datasets, e.g., data from an unnamed social network in [22], crowdsourced voice commands in [18], and proprietary data by Huawei in [5]; and (3) realistic federated datasets that are derived from publicly available data, but which are not…”
FedSpeech: Federated Text-to-Speech with Continual Learning (2021, background): “After the concept of federated learning is proposed by Google recently [Koneˇcn`y et al. , 2016], most federated learning researches focus on computer vision [McMahan et al. , 2017], natural language processing [Hard et al. , 2018], and voice recognition [Leroy et al. , 2019].”
A Survey on federated learning* (2020, background): “Study of FL can be divided into two main areas: the first one is the extension of Google’s work on decentralized model for distributed mobile devices such as prediction on keyboard [49], and service placement for mobile edge computing [50].”
Communication-Efficient Federated Learning for Wireless Edge Intelligence in IoT (2020, unknown): “However, the way we use Adam is different in our work from [18]: our method uses Adam optimisation at clients, and the Adam values are also aggregated at the server, as opposed to using Adam solely at the server.”
Edge Learning for B5G Networks With Distributed Signal Processing: Semantic Communication, Edge Computing, and Wireless Sensing (2022, unknown): “A typical use case of HFL is voice assistants for smart homes [54], in which”
Federated Learning for Healthcare Informatics (2019, background): “rediction in a mobile keyboard [97], and learning out-of-vocabulary (OOV) words for the purpose of expanding the vocabulary of a virtual keyboard for smartphones [21]. Except for the text data, Leroy [77] investigated the use of federated learning on crowd-sourced speech data, to solve out-of-domain”
Training Keyword Spotting Models on Non-IID Data with Federated Learning (2020, background): “Recently, the feasibility of training keyword-spotting algorithms with FL has been explored with smaller datasets [19].”
Federated Neural Architecture Search (2020, background): “Such application scenarios are abounding: next-word prediction [16, 30], speech keyword spotting [22], image classification [27], etc.”
A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, unknown): “Another interesting approach to training high capacity models having fewer parameters is discussed in [23].”
Asynchronous Online Federated Learning for Edge Devices with Non-IID Data (2019, background): “, FedAvg [1] and its extensions [2]–[6]), where at each global Device 1 Central Server”
FetchSGD: Communication-Efficient Federated Learning with Sketching (2020, unknown): “These devices have data that is useful for training a variety of models – for text prediction, speech modeling, facial recognition, document identification, and other tasks (Shi et al., 2016; Brisimi et al., 2018; Leroy et al., 2019; Tomlinson et al., 2009).”
A Survey on Federated Learning: The Journey From Centralized to Distributed On-Site Learning and Beyond (2021, background): “The work in [48] targets the practical FL usage for speech data when detecting wake word, such as “Hey Siri” and “OK Google.”
A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, background): “Wake-word recognition [98], such as ‘Hey Siri’ and ‘OK Google’, is a typical application of horizontal partition because each user speaks the same sentence with a different voice.”