Stochastic, Distributed and Federated Optimization for Machine Learning

By Jakub Konecný. 2017

38 citations recorded by the source corpus.

We study optimization algorithms for the finite sum problems frequently arising in machine learning applications. First, we propose novel variants of stochastic gradient descent with a variance reduction property that enables linear convergence for strongly convex objectives. Second, we study distributed setting, in which the data describing the optimization problem does not fit into a single computing node. In this case, traditional methods are inefficient, as the communication costs inherent in distributed optimization become the bottleneck. We propose a communication-efficient framework which iteratively forms local subproblems that can be solved with arbitrary local optimization algorithms. Finally, we introduce the concept of Federated Optimization/Learning, where we try to solve the machine learning problems without having data stored in any centralized manner. The main motivation comes from industry when handling user-generated data. The current prevalent practice is that companies collect vast amounts of user data and store them in datacenters. An alternative we propose is not to collect the data in first place, and instead occasionally use the computational power of users' devices to solve the very same optimization problems, while alleviating privacy concerns at the same time. In such setting, minimization of communication rounds is the primary goal, and we demonstrate that solving the optimization problems in such circumstances is conceptually tractable.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Optimization & Training, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Optimization & Training

In Federated & Distributed Learning

Citations in the corpus

Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “Federated Learning (FL), which has a central parameter server, enables massive edge computing devices to jointly learn a centralized model while keeping all local data localized [13, 34, 26, 12, 18, 30, 57].”
Data Encoding for Byzantine-Resilient Distributed Optimization (2019, background): “In several applications of distributed learning, including the Internet of Battleﬁeld Things (IoBT) [A + 18], federated optimization [Kon17], the recruited worker nodes might be partially trusted with their computation.”
QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding (2016, methodology): “Our use case is different from the federated learning application of [38, 24], and has the advantage of being more efﬁcient to compute on a GPU.”
Byzantine-Resilient High-Dimensional SGD with Local Iterations on Heterogeneous Data (2020, unknown): “In the federated learning (FL) paradigm (Konecný, 2017; Konecný et al., 2016; McMahan et al., 2017; Mohri et al., 2019), several clients (e.”
Data Encoding for Byzantine-Resilient Distributed Gradient Descent (2018, background): “In several applications of distributed learning, including the Internet of Battlefield Things (IoBT) [4], federated optimization [5], the recruited worker nodes might be partially trusted with their computation.”
Byzantine-Tolerant Distributed Coordinate Descent (2019, background): “With growing interest in federated learning [2], harnessing of non-enterprise nodes in the computation is being envisaged.”
Data Encoding Methods for Byzantine-Resilient Distributed Optimization (2019, background): “In several applications of distributed learning, including the Internet of Battleﬁeld Things (IoBT) [3], federated optimization [4], the recruited worker nodes might be partially trusted with their computation.”