Federated Learning of N-Gram Language Models

By Mingqing Chen, Ananda Theertha Suresh, Rajiv Mathews, Adeline Wong, Cyril Allauzen, Franccoise Beaufays, Michael D. Riley. 2019

80 citations recorded by the source corpus.

We propose algorithms to train production-quality n-gram language models using federated learning. Federated learning is a distributed computation platform that can be used to train global models for portable devices such as smart phones. Federated learning is especially relevant for applications handling privacy-sensitive data, such as virtual keyboards, because training is performed without the users’ data ever leaving their devices. While the principles of federated learning are fairly generic, its methodology assumes that the underlying models are neural networks. However, virtual keyboards are typically powered by n-gram language models for latency reasons. We propose to train a recurrent neural network language model using the decentralized FederatedAveraging algorithm and to approximate this federated model server-side with an n-gram model that can be deployed to devices for fast inference. Our technical contributions include ways of handling large vocabularies, algorithms to correct capitalization errors in user data, and efficient finite state transducer algorithms to convert word language models to word-piece language models and vice versa. The n-gram language models trained with federated learning are compared to n-grams trained with traditional server-based algorithms using A/B tests on tens of millions of users of a virtual keyboard. Results are presented for two languages, American English and Brazilian Portuguese. This work demonstrates that high-quality n-gram language models can be trained directly on client mobile devices without sensitive training data ever leaving the devices.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

In Federated & Distributed Learning

Citations in the corpus

FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “Promising application domains include, but are not limited to, computer vision [77, 78], natural language processing [79, 80, 81, 82, 83], ﬁnance [74, 35, 84], transportation [85, 86, 87, 88, 89, 90, 91, 92, 93, 94], digital health [95, 96, 97, 98, 99, 100, 101], recommendation [102, 103, 104, 105,…”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “Several current applications of FL include text prediction in mobile device messaging [1, 2, 3, 4, 5], speech recognition [6], face recognition for device access [7, 8], and maintaining decentralized predictive models across health organizations [9, 10, 11].”
On the Outsized Importance of Learning Rates in Local Update Methods (2020, background): “A particularly influential recent approach is model-agnostic meta-learning (MAML) proposed by Finn et al. (2017). The core idea has inspired a number of extensions (Antoniou et al.”
SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning (2019, background): “…(Hard et al., 2018; Yang et al., 2018), emoji prediction (Ramaswamy et al., 2019), decoder models (Chen et al., 2019b), vocabulary estimation (Chen et al., 2019a), low latency vehicle-to-vehicle communication (Samarakoon et al., 2018), and predictive models in health (Brisimi et al., 2018).”
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning (2019, background): “(Hard et al., 2018; Yang et al., 2018), emoji prediction (Ramaswamy et al., 2019), decoder models (Chen et al., 2019b), vocabulary estimation (Chen et al., 2019a), low latency vehicle-to-vehicle communication (Sama-rakoon et al., 2018), and predictive models in health (Brisimi et al., 2018).”
FedBoost: Communication-Efficient Algorithms for Federated Learning (2020, unknown): “…(Hard et al., 2018; Yang et al., 2018), emoji prediction (Ramaswamy et al., 2019), decoder models (Chen et al., 2019b), vocabulary estimation (Chen et al., 2019a), low latency vehicleto-vehicle communication (Samarakoon et al., 2018), and predictive models in health (Brisimi et al., 2018).”
Training Keyword Spotting Models on Non-IID Data with Federated Learning (2020, methodology): “Multiple production models have been trained with federated learning, including next-word prediction [15], emoji prediction [16], n-gram language models [17], and query suggestions [18] for mobile keyboards.”