Federated Optimization: Distributed Machine Learning for On-Device Intelligence

By Jakub Konecný, H. Brendan McMahan, Daniel Ramage, Peter Richtárik. 2016

2,231 citations recorded by the source corpus.

We introduce a new and increasingly relevant setting for distributed optimization in machine learning, where the data defining the optimization are unevenly distributed over an extremely large number of nodes. The goal is to train a high-quality centralized model. We refer to this setting as Federated Optimization. In this setting, communication efficiency is of the utmost importance and minimizing the number of rounds of communication is the principal goal. A motivating example arises when we keep the training data locally on users' mobile devices instead of logging it to a data center for training. In federated optimziation, the devices are used as compute nodes performing computation on their local data in order to update a global model. We suppose that we have extremely large number of devices in the network --- as many as the number of users of a given service, each of which has only a tiny fraction of the total data available. In particular, we expect the number of data points available locally to be much smaller than the number of devices. Additionally, since different users generate data with different patterns, it is reasonable to assume that no device has a representative sample of the overall distribution. We show that existing algorithms are not suitable for this setting, and propose a new algorithm which shows encouraging experimental results for sparse convex problems. This work also sets a path for future research needed in the context of \federated optimization.

Subfields: Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Federated Learning: Strategies for Improving Communication Efficiency (2016, background): “Recent works show that a careful choice of the server-side learning rate can lead to faster convergence [13, 12, 10].”
Training Neural Networks with Fixed Sparse Masks (2021, background): “An extreme variant of distributed training is federated learning [22, 35].”
Federated Learning: Collaborative Machine Learning without Centralized Training Data (2022, positive): “A detailed survey of algorithms for solving the optimization problem (1) is given in [14].”
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, background): “Federated learning has become an important paradigm in large-scale machine learning where the training data remains distributed over a large number of clients, which may be mobile phones or network sensors [20, 19, 23, 24, 13].”
Federated Learning and Differential Privacy: Software tools analysis, the Sherpa.ai FL framework and methodological guidelines for preserving data privacy (2020, background): “in Section5.4as methodological guidelines for preserving data privacy with FL using Sherpa.aiFL. 5.1. Key aspects of distributed computing for Federated Machine Learning The recent introduction of FL [26,14,7] responds to the need for novel distributed machine learning algorithms in a setting that c”
rTop-k: A Statistical Estimation Approach to Distributed SGD (2020, background): “, one iteration over the local data adapting the federated setting in [7], [8]).”
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates (2020, background): “In efforts to depart from the traditional parameter server paradigm (Dean et al., 2012), federated learning (Konečnỳ et al., 2016; McMahan *Equal contribution 1EPFL, Lausanne, Switzerland 2Mila and DIRO, Université de Montréal, Canada.”
Distributed Fixed Point Methods with Compressed Iterates (2019, background): “This is particularly true in federated learning [6,26,27,33], where a network of a very large number ( n ) of computing devices, such as mobile phones or hospitals, is required to jointly train a machine learning model from the private data stored on these devices, and more so if these models are…”
Communication Efficient Decentralized Training with Multiple Local Updates (2019, methodology): “Currently the state-of-the-art algorithm in federated optimization is Federated Averaging (FedAvg) [14, 26], which is a centralized optimization method.”
Improving Federated Learning Personalization via Model Agnostic Meta Learning (2019, methodology): “Successful algorithmic extensions to the central idea include training a differential private model (McMahan et al., 2018), compression (Koneˇcn´y et al., 2016b; Caldas et al., 2018a), secure aggregation (Bonawitz et al., 2017), and a smaller number of always-participating nodes (Yang et al., 2019).”
Gradient Descent with Compressed Iterates (2019, background): “The training of high-dimensional federated learning models [16, 17] reduces to solving an optimization problem of the form x ̊ “ argmin xPRd «”
Decentralized Bayesian Learning over Graphs (2019, neutral): “We focus on star topology since federated learning methods [6, 8, 7] (only) consider networks with this structure.”
Incentive Design for Efficient Federated Learning in Mobile Networks: A Contract Theory Approach (2019, background): “Given the target local accuracy, the number of local iterations is log( 1 εn ) [4], [11].”
Semi-Cyclic Stochastic Gradient Descent (2019, background): “But in some applications, such as in Federated Learning (Konečný et al., 2016; McMahan et al., 2017) or when running SGD in “real time”, some cyclic patterns in the data samples are hard to avoid.”
Towards Federated Learning at Scale: System Design (2019, methodology): “The general description of FL has been given by McMahan & Ramage (2017), and its theory has been explored in Kone ˇ cn ´ y et al. (2016a); McMahan et al. (2017; 2018).”
Federated Machine Learning (2019, methodology): “ investigating methods for data integration that is compliant with data privacy and security laws. 2 AN OVERVIEW OF FEDERATED LEARNING The concept of federated learning is proposed by Google recently [36, 37, 41]. Their main idea is to build machine learning models based on data sets that are distri”