Federated Learning Of Out-Of-Vocabulary Words

By Mingqing Chen, Rajiv Mathews, Tom Y. Ouyang, Franccoise Beaufays. 2019

182 citations recorded by the source corpus.

We demonstrate that a character-level recurrent neural network is able to learn out-of-vocabulary (OOV) words under federated learning settings, for the purpose of expanding the vocabulary of a virtual keyboard for smartphones without exporting sensitive text to servers. High-frequency words can be sampled from the trained generative model by drawing from the joint posterior directly. We study the feasibility of the approach in two settings: (1) using simulated federated learning on a publicly available non-IID per-user dataset from a popular social networking website, (2) using federated learning on data hosted on user mobile devices. The model achieves good recall and precision compared to ground-truth OOV words in setting (1). With (2) we demonstrate the practicality of this approach by showing that we can learn meaningful OOV words with good character-level prediction accuracy and cross entropy loss.

Subfields: Artificial Intelligence, Natural Language Processing, Federated & Distributed Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Federated & Distributed Learning

In Signal Processing

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “Google makes extensive use of federated learning in the Gboard mobile keyboard [323, 196, 420, 98, 329], as well as in features on Pixel phones [18] and in Android Messages [375].”
SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning (2019, background): “…(Hard et al., 2018; Yang et al., 2018), emoji prediction (Ramaswamy et al., 2019), decoder models (Chen et al., 2019b), vocabulary estimation (Chen et al., 2019a), low latency vehicle-to-vehicle communication (Samarakoon et al., 2018), and predictive models in health (Brisimi et al., 2018).”
Real-World Image Datasets for Federated Learning (2019, background): “[Chen et al. , 2019] further improves next-word-prediction by addressing the out-of-vocabulary problem in language model personalization.”
Privacy Preserving Machine Learning with Homomorphic Encryption and Federated Learning (2021, background): “Besides this, natural language processing [42] and recommendation systems [43] are also applicable for federated learning as well.”
Self-supervised Cross-silo Federated Neural Architecture Search (2021, methodology): “Google has applied FL to its Gboard mobile keyboard applications [28]–[30].”
A Survey on Federated Learning: The Journey From Centralized to Distributed On-Site Learning and Beyond (2021, background): “[64] adapted FL, more precisely federated character-based recurrent”
FedScale: Benchmarking Model and System Performance of Federated Learning at Scale (2021, methodology): “It enables model training and evaluation on end-user data, while circumventing high cost and privacy risks in gathering the raw data from clients, with applications in diverse domains: for example, NVIDIA applies FL to create medical imaging AI [37]; Google runs federated training of NLP models in G”
Gradient-Leaks: Understanding and Controlling Deanonymization in Federated Learning (2018, background): “At this point, we remark that research towards a Federated Learning system encompasses among many other things, architecture [15], optimization techniques [46, 54], strategies to improve communication [47], aggregation [16], implementation [1], and applications [21,35,83].”
FedScale: Benchmarking Model and System Performance of Federated Learning (2021, methodology): “It enables model training and evaluation on end-user data, while circumventing high cost and privacy risks in gathering the raw data from clients, with applications in diverse domains: for example, NVIDIA applies FL to create medical imaging AI [26]; Google runs federated training of NLP models in G”
A Survey on federated learning* (2020, background): “These includes cases in industry applications [2][3] or mobile devices [4] that data are not available to be aggregated with legal concern.”
FedBoost: Communication-Efficient Algorithms for Federated Learning (2020, unknown): “…(Hard et al., 2018; Yang et al., 2018), emoji prediction (Ramaswamy et al., 2019), decoder models (Chen et al., 2019b), vocabulary estimation (Chen et al., 2019a), low latency vehicleto-vehicle communication (Samarakoon et al., 2018), and predictive models in health (Brisimi et al., 2018).”