A Framework for Parallel and Distributed Training of Neural Networks
By Simone Scardapane, P. Lorenzo. 2016
27 citations recorded by the source corpus.
The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification-decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach.
Subfields: Artificial Intelligence, ML Systems & Hardware
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mingyi Hong
- Davood Hajinezhad
- Mingmin Zhao
- Simone Scardapane
- Michele Scarpiniti
- Aurelio Uncini
- Enzo Baccarelli
- Haoran Sun
In ML Systems & Hardware
Citations in the corpus
- Why Should We Add Early Exits to Neural Networks? (2020, background): “An interesting question concerns the additional implementation of a distributed training phase [57], which is necessary whenever the separate tiers have to adapt to new data without the possibility of communicating everything to a centralized controller.”
- Prox-PDA: The Proximal Primal-Dual Algorithm for Fast Distributed Nonconvex Optimization and Learning Over Networks (2017, background): “Recent works that apply distributed optimization algorithms to machine learning applications include (Scardapane et al., 2016; Aybat & Hamedani, 2016; Scardapane & Lorenzo, 2016).”