Federated Optimization: Distributed Optimization Beyond the Datacenter

By Jakub Konecný, H. Brendan McMahan, Daniel Ramage. 2015

823 citations recorded by the source corpus.

We introduce a new and increasingly relevant setting for distributed optimization in machine learning, where the data defining the optimization are distributed (unevenly) over an extremely large number of \nodes, but the goal remains to train a high-quality centralized model. We refer to this setting as Federated Optimization. In this setting, communication efficiency is of utmost importance. A motivating example for federated optimization arises when we keep the training data locally on users' mobile devices rather than logging it to a data center for training. Instead, the mobile devices are used as nodes performing computation on their local data in order to update a global model. We suppose that we have an extremely large number of devices in our network, each of which has only a tiny fraction of data available totally; in particular, we expect the number of data points available locally to be much smaller than the number of devices. Additionally, since different users generate data with different patterns, we assume that no device has a representative sample of the overall distribution. We show that existing algorithms are not suitable for this setting, and propose a new algorithm which shows encouraging experimental results. This work also sets a path for future research needed in the context of federated optimization.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “However, it will always be the case that there are applications and scenarios where using a smaller or less expensive model is helpful, for example when performing client-side inference or federated learning [Konečnỳ et al., 2015, 2016].”
Federated Learning: Strategies for Improving Communication Efficiency (2016, background): “Recent works show that a careful choice of the server-side learning rate can lead to faster convergence [12, 11, 9].”
Decentralized gradient methods: does topology matter? (2020, neutral): “This plot warns against extending the empirical ﬁnding in (Lian et al., 2017, 2018; Luo et al., 2019) to settings where local datasets can be highly diﬀerent as it can be for example in the case of federated learning (Konecn´y et al., 2015).”
Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “Federated Learning (FL), which has a central parameter server, enables massive edge computing devices to jointly learn a centralized model while keeping all local data localized [13, 34, 26, 12, 18, 30, 57].”
Central Server Free Federated Learning over Single-sided Trust Social Networks (2019, background): “Early examples of research into federated learning also include Konen et al. (2015, 2016), and a widespread blog article posted by Google AI (McMahan and Ramage, 2017).”
Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “Some references are Konecný et al. (2015, 2016); McMahan et al. (2016).”
Practical Distributed Learning: Secure Machine Learning with Communication-Efficient Local Updates (2019, methodology): “Existing federated optimization methods (Konevcnỳ et al., 2015; 2016; McMahan et al., 2016) unfortuntaely lack provable convergence guarantees.”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “For training in a massively distributed scenario, the work of [26, 27, 28] introduces the setting of federated learning.”
COLA: Decentralized Linear Learning (2018, methodology): “Federated learning was first described by [12, 13, 19] as decentralized learning for on-device learning applications, combining a global shared model with local personalized models.”
DRACO: Byzantine-resilient Distributed Training via Redundant Gradients (2018, background): “This work shares ideas with federated learning, in which training is distributed among a large number of compute nodes without centralized training data (Konečnỳ et al., 2015; 2016; Bonawitz et al., 2016).”
Federated Multi-Task Learning (2017, methodology): “Work on federated learning thus far has been limited to the study of fully global models [25, 36, 26].”
Distributed Statistical Machine Learning in Adversarial Settings (2017, methodology): “To address privacy concerns, a new machine learning paradigm called Federated Learning was proposed by Google researchers [16, 1].”
LoAdaBoost: Loss-based AdaBoost federated machine learning with reduced computational complexity on IID and non-IID intensive care data (2018, background): “[6] introduced a federated optimization algorithm suitable for training massively distributed, non-identically independently distributed (non-IID) and unbalanced datasets.”
Federated Learning with Non-IID Data (2018, background): “Federated learning [3, 4, 5] offers an approach to do training locally as well.”
AIDE: Fast and Communication Efficient Distributed Optimization (2016, methodology): “6 Connection to a Practical Distributed Version of SVRG In this section, we discuss the connection between SGD, INEXACTDANE and a version of distributed SVRG that was observed to perform well in practice [7], but has not yet been successfully analyzed.”