APPLIED FEDERATED LEARNING: IMPROVING GOOGLE KEYBOARD QUERY SUGGESTIONS

By Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, Franccoise Beaufays. 2018

715 citations recorded by the source corpus.

Federated learning is a distributed form of machine learning where both the training data and model training are decentralized. In this paper, we use federated learning in a commercial, global-scale setting to train, evaluate and deploy a model to improve virtual keyboard search suggestion quality without direct access to the underlying user data. We describe our observations in federated training, compare metrics to live deployments, and present resulting quality increases. In whole, we demonstrate how federated learning can be applied endto-end to both improve user experiences and enhance user privacy.

Subfields: Artificial Intelligence, Computer Science Applications, Sociology and Political Science, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Sociology and Political Science

In Efficiency & Compression

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “, 2020b) while improving data privacy (Hard et al., 2018; Yang et al., 2018; Ramaswamy et al., 2019; Huba et al., 2022), yet its impact in terms of carbon emission is often overlooked.”
Advances and Open Problems in Federated Learning (2019, background): “mobile keyboard [10]–[14], as well as in features on Pixel phones and in Android Messages [15].”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “Several current applications of FL include text prediction in mobile device messaging [1, 2, 3, 4, 5], speech recognition [6], face recognition for device access [7, 8], and maintaining decentralized predictive models across health organizations [9, 10, 11].”
On the Outsized Importance of Learning Rates in Local Update Methods (2020, neutral): “There is enough evidence that these methods are useful in practice in complex scenarios (Hard et al., 2018; Yang et al., 2018; Hard et al., 2020), yet on a theoretical level, many works derive results comparable to, or worse than, that of mini-batch SGD in heterogeneous or even homogeneous…”
Three Approaches for Personalization with Applications to Federated Learning (2020, positive): “This global model beneﬁts from having access to client data and can often perform better on several learning problems, including next word prediction (Hard et al., 2018; Yang et al., 2018) and predictive models in health (Brisimi et al., 2018).”
Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning (2020, background): “A recent promising implementation of federated learning, as well as its application to Google keyboard query suggestions is demonstrated in [8].”
Robust Aggregation for Federated Learning (2019, background): “The paradigm has found widespread applications ranging from mobile apps deployed on millions of devices [6, 87], to sensitive healthcare applications [40, 73].”
Generative Models for Effective ML on Private, Decentralized Datasets (2019, background): “…might inspect misclassified examples to look for issues in the features or 1Despite these limitations, FL systems (Bonawitz et al., 2019) have proven effective for training and deploying real-world models, for example see Hard et al. (2018); Yang et al. (2018); Chen et al. (2019). labels (T2).”
Secure Federated Submodel Learning (2019, neutral): “Google has deployed federated learning in its Android keyboard, called Gboard, to polish several language tasks, including nextword prediction [2], query suggestion [38], out-of-vocabulary words learning [11], and emoji prediction [12].”
SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning (2019, background): “…all client data and can often result in a beneﬁcial performance, as reported in several tasks, including next word prediction (Hard et al., 2018; Yang et al., 2018), emoji prediction (Ramaswamy et al., 2019), decoder models (Chen et al., 2019b), vocabulary estimation (Chen et al., 2019a), low…”
Federated Learning for Emoji Prediction in a Mobile Keyboard (2019, positive): “This work builds on previous practical applications of federated learning in Yang et al. (2018); Hard et al. (2018); Bonawitz et al. (2019).”
Federated Learning Of Out-Of-Vocabulary Words (2019, methodology): “Our work builds upon recent advances in Federated Learning (FL) (Koneˇcn`y et al., 2016; Yang et al., 2018; Hard et al., 2018), a decentralized learning technique to train models such as neural networks on users’ devices, uploading only ephemeral model updates to the server for aggregation, and…”
Towards Federated Learning at Scale: System Design (2019, methodology): “This approach has been taken by Google’s Gboard mobile keyboard team, using our FL system (Yang et al., 2018).”
Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System (2019, background): “Classical studies based on federated averaging used CNNs to train on benchmark image recognition tasks [3], and LSTM on a language modeling tasks [19, 20].”
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning (2019, background): “in including next word prediction (Hard et al., 2018; Yang et al., 2018), emoji prediction (Ramaswamy et al.”
FetchSGD: Communication-Efficient Federated Learning with Sketching (2020, neutral): “, 2019), and is the most commonly used optimization algorithm in the federated setting (Yang et al., 2018).”
Optimizing Federated Learning on Non-IID Data with Reinforcement Learning (2020, background): “[3] developed a secure aggregation protocol that enables a server to perform the computation of high-dimensional data from the devices, which is widely deployed in production environments [4, 17].”
A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, neutral): “Recently, Google has invented an FL-based keyboard suggestion [29] by training the model on-device of the user and aggregate the model information in the server.”
Federating Recommendations Using Differentially Private Prototypes (2020, background): “Federated learning has proved eﬃcient in learning deep neural networks for image classiﬁcation (McMahan et al., 2017) and text generation tasks (Yang et al., 2018; Hard et al., 2018).”