Federated Learning for Emoji Prediction in a Mobile Keyboard

By Swaroop Indra Ramaswamy, Rajiv Mathews, Kanishka Rao, Franccoise Beaufays. 2019

334 citations recorded by the source corpus.

We show that a word-level recurrent neural network can predict emoji from text typed on a mobile keyboard. We demonstrate the usefulness of transfer learning for predicting emoji by pretraining the model using a language modeling task. We also propose mechanisms to trigger emoji and tune the diversity of candidates. The model is trained using a distributed on-device learning framework called federated learning. The federated model is shown to achieve better performance than a server-trained model. This work demonstrates the feasibility of using federated learning to train production-quality models for natural language understanding tasks while keeping users' data on their devices.

Subfields: Education, Natural Language Processing, Efficiency & Compression, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Education

In Natural Language Processing

In Efficiency & Compression

In Human-Computer Interaction

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “On-Device Learning On-device AI is becoming more ubiquitously adopted to enable model personalization [82], [83], [84] while improving data privacy [85], [86], [87], yet its impact in terms of carbon emission is often overlooked.”
Advances and Open Problems in Federated Learning (2019, background): “Google makes extensive use of federated learning in the Gboard mobile keyboard [323, 196, 420, 98, 329], as well as in features on Pixel phones [18] and in Android Messages [375].”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “Several current applications of FL include text prediction in mobile device messaging [1, 2, 3, 4, 5], speech recognition [6], face recognition for device access [7, 8], and maintaining decentralized predictive models across health organizations [9, 10, 11].”
SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning (2019, background): “…in a beneﬁcial performance, as reported in several tasks, including next word prediction (Hard et al., 2018; Yang et al., 2018), emoji prediction (Ramaswamy et al., 2019), decoder models (Chen et al., 2019b), vocabulary estimation (Chen et al., 2019a), low latency vehicle-to-vehicle…”
Federated Learning: Challenges, Methods, and Future Directions (2019, background): “By jointly learning user behavior across a large pool of mobile phones, statistical models can power applications such as next-word prediction, face detection, and voice recognition [46, 89].”
Overcoming Forgetting in Federated Learning on Non-IID Data (2019, neutral): “distribution [2].”
Federated Class-Incremental Learning (2022, methodology): “Recently, it has attracted signiﬁcant interests in academia and achieved remarkable successes in various industrial applications, e.g. , autonomous driving [39], wearable devices [33], medical diagnosis [10,52] and mobile phones [36].”
A Survey on Federated Learning: The Journey From Centralized to Distributed On-Site Learning and Beyond (2021, background): “Ramaswamy et al. [63] have proved the ability of the recurrent neural network to predict emoji from text on Gboard through FL. Chen et al. [64] adapted FL, more precisely federated character-based recurrent neural network, to learn out-of-vocabulary (OOV) words.”
A review of applications in federated learning (2020, methodology): “Besides, emoji prediction is also a research hotspot (Ramaswamy et al.,2019).”
An Overview of Federated Deep Learning Privacy Attacks and Defensive Strategies (2020, background): “Researchers demonstrated an implementation for next word prediction on mobile devices [4], [5] as well as emoticon prediction [6] among others.”
Eavesdrop the Composition Proportion of Training Labels in Federated Learning (2019, background): “, human activity recognition [1], heart rate monitoring [4] and keyboard prediction [2, 3]), but also in other fields, such as healthcare industry (e.”