Federated Learning for Mobile Keyboard Prediction

By Andrew Straiton Hard, Kanishka Rao, Rajiv Mathews, Franccoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, Daniel Ramage. 2018

1,848 citations recorded by the source corpus.

We train a recurrent neural network language model using a distributed, on-device learning framework called federated learning for the purpose of next-word prediction in a virtual keyboard for smartphones. Server-based training using stochastic gradient descent is compared with training on client devices using the Federated Averaging algorithm. The federated algorithm, which enables training on a higher-quality dataset for this use case, is shown to achieve better prediction recall. This work demonstrates the feasibility and benefit of training language models on client devices without exporting sensitive user data to servers. The federated learning environment gives users greater control over the use of their data and simplifies the task of incorporating privacy by default with distributed training and aggregation across a population of client devices.

Subfields: Artificial Intelligence, Information Systems, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Optimization & Training

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “, 2020b) while improving data privacy (Hard et al., 2018; Yang et al., 2018; Ramaswamy et al., 2019; Huba et al., 2022), yet its impact in terms of carbon emission is often overlooked.”
Advances and Open Problems in Federated Learning (2019, background): “Google makes extensive use of federated learning in the Gboard mobile keyboard [323, 196, 420, 98, 329], as well as in features on Pixel phones [18] and in Android Messages [375].”
How To Backdoor Federated Learning (2018, background): “ferent from publicly available datasets [25].”
The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation (2021, background): “Figure 3 scales up to n = 1000 clients on SO-NWP (similar to production settings described in Hard et al. (2018); Ramaswamy et al. (2019)) and shows the validation accuracies during training across different noise multipliers.7 We set c = 1 and ηs = 1 for z ≈ 0.3 and z ≈ 0.5 and ηs = 3 otherwise. z…”
Flower: A Friendly Federated Learning Research Framework (2020, background): “While closed production-grade systems report client numbers in the thousands or even millions (Hard et al., 2019), few research papers use populations of more than 100 clients, as can be seen in Figure 1.”
FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “Promising application domains include, but are not limited to, computer vision [77, 78], natural language processing [79, 80, 81, 82, 83], ﬁnance [74, 35, 84], transportation [85, 86, 87, 88, 89, 90, 91, 92, 93, 94], digital health [95, 96, 97, 98, 99, 100, 101], recommendation [102, 103, 104, 105,…”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “Several current applications of FL include text prediction in mobile device messaging [1, 2, 3, 4, 5], speech recognition [6], face recognition for device access [7, 8], and maintaining decentralized predictive models across health organizations [9, 10, 11].”
On the Outsized Importance of Learning Rates in Local Update Methods (2020, background): “There is enough evidence that these methods are useful in practice in complex scenarios (Hard et al., 2018; Yang et al., 2018; Hard et al., 2020), yet on a theoretical level, many works derive results comparable to, or worse than, that of mini-batch SGD in heterogeneous or even homogeneous…”
Personalized Federated Learning with Moreau Envelopes (2020, methodology): “Regarding using a model in different contexts, in the next-character prediction task in [29], the requirement to predict differently among devices raises a need to inspect more features about the context of client devices during training, which was studied in [30].”
Three Approaches for Personalization with Applications to Federated Learning (2020, background): “This global model beneﬁts from having access to client data and can often perform better on several learning problems, including next word prediction (Hard et al., 2018; Yang et al., 2018) and predictive models in health (Brisimi et al., 2018).”
Salvaging Federated Learning by Local Adaptation (2020, background): “5 billion sentences from 1.5 million North American participants (Hard et al., 2018).”
Generative Models for Effective ML on Private, Decentralized Datasets (2019, methodology): “Often in FL, labels can be inferred naturally from context on device—for example, the next-word predictor in a keyboard app (e.g., as in Hard et al. (2018)) is a problem where the input is a sequence of words and the label is the subsequent word.”
Secure Federated Submodel Learning (2019, methodology): “Google has deployed federated learning in its Android keyboard, called Gboard, to polish several language tasks, including nextword prediction [2], query suggestion [38], out-of-vocabulary words learning [11], and emoji prediction [12].”
Federated Evaluation of On-device Personalization (2019, methodology): “The network architecture of the next-word prediction model is described in Hard et al. (2018).”
SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning (2019, background): “…model beneﬁts from all client data and can often result in a beneﬁcial performance, as reported in several tasks, including next word prediction (Hard et al., 2018; Yang et al., 2018), emoji prediction (Ramaswamy et al., 2019), decoder models (Chen et al., 2019b), vocabulary estimation (Chen et…”
Real-World Image Datasets for Federated Learning (2019, methodology): “Federated learning algorithms are also applied to industrial scenarios including next-word-prediction [Hard et al. , 2018], speech keyword spot-ting [Leroy et al. , 2018], and images classiﬁcation [Liu et al. , 2018].”
Improving Federated Learning Personalization via Model Agnostic Meta Learning (2019, methodology): “The basic FL algorithm, Federated Averaging (FedAvg) (McMahan et al., 2017), has been used in production applications, for instance for next word prediction in mobile keyboard (Hard et al., 2018), which shows that Federated Learning can outperform the best model trained in a dat-acenter.”
First Analysis of Local GD on Heterogeneous Data (2019, methodology): “This algorithm has been used in several machine learning applications such as mobile keyboard prediction [5], and strategies for improving its communication efﬁciency were explored in [7].”
AdaCliP: Adaptive Clipping for Private SGD (2019, background): “Most state of the art machine learning models are trained on user data, examples include keyboard models [6], automatic video transcription [7] among others.”