Training Neural Networks Without Gradients: A Scalable ADMM Approach

By Gavin Taylor, Ryan Burmeister, Zheng Xu, Bharat Singh, Ankit B. Patel, Tom Goldstein. 2016

297 citations recorded by the source corpus.

With the growing importance of large network models and enormous training datasets, GPUs have become increasingly necessary to train neural networks. This is largely because conventional optimization algorithms rely on stochastic gradient methods that don't scale well to large numbers of cores in a cluster setting. Furthermore, the convergence of all gradient methods, including batch methods, suffers from common problems like saturation effects, poor conditioning, and saddle points. This paper explores an unconventional training method that uses alternating direction methods and Bregman iteration to train networks without gradient descent steps. The proposed method reduces the network training problem to a sequence of minimization substeps that can each be solved globally in closed form. The proposed method is advantageous because it avoids many of the caveats that make gradient methods slow on highly nonconvex problems. The method exhibits strong scaling in the distributed setting, yielding linear speedups even when split over thousands of cores.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Training Neural Networks Using Features Replay (2018, methodology): “Similarly, [34] used Lagrange multipliers to enforce equality constraints between auxiliary variables and activations.”
Decoupled Neural Interfaces using Synthetic Gradients (2016, background): “Other works such as (Taylor et al., 2016; Carreira-Perpinán & Wang, 2014) allow training of layers in parallel without backpropagation, but in practice are not scalable to more complex and generic network architectures.”
DAGs with NO TEARS: Continuous Optimization for Structure Learning (2018, methodology): “…deep neural networks are often learned using various descendants of stochastic gradient descent (SGD) (Bousquet and Bottou, 2008; Kingma and Ba, 2014; Bottou et al., 2016), although recent work has proposed other techniques such as ADMM (Taylor et al., 2016) and Gauss-Newton (Botev et al., 2017).”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “onsolidation Ensemble Learning[114,148,217] Knowledge Distillation[10,96] Model Averaging: Direct[20,33,168,266], Elastic[121,155,248,258], Natural Gradient[11,195] Optimization Algorithms First-Order[43,123,126,141,145,163,227] Second-Order[11,28,56,94,133,165,172] Evolutionary[169,192,234,242] Hyp”
Adaptive Consensus ADMM for Distributed Optimization (2017, background): “…involving a composite objective f ( v ) = i f i ( v ) , where worker i stores the data needed to compute f i , and so is well suited for distributed model ﬁtting problems (Boyd et al., 2011; Zhang & Kwok, 2014; Song et al., 2016; Chang et al., 2016; Goldstein et al., 2016; Taylor et al., 2016).”
You Only Propagate Once: Painless Adversarial Training Using Maximal Principle (2019, background): “Different ways have been proposed to decouple the sequential process to parallelize it, including ADMM [32], synthetic gradients [33], delayed gradient [34], lifted machines [35–37].”
Ensemble-Compression: A New Method for Parallel Training of Deep Neural Networks (2016, background): “Actually, MA (and its variants such as ADMM [22, 20]) was originally proposed for convex optimization.”
Decoupled Parallel Backpropagation with Convergence Guarantee (2018, background): “In (CarreiraPerpinan & Wang, 2014; Taylor et al., 2016), the authors loosened the exact connections between layers by introducing auxiliary variables.”
Associated Learning: Decomposing End-to-End Backpropagation Based on Autoencoders and Target Propagation (2019, background): “Thus, many studies have suggested replacing backpropagation by more biologically plausible methods or by gradient-free methods [28] in the hope of decreasing the computational time and memory consumption [27, 29, 30].”
You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle (2019, methodology): “This includes ADMM [34], synthetic gradients [14], delayed gradient [12], lifted machines [1,9,19].”
Decoupled Greedy Learning of CNNs (2019, methodology): “Another line of related work inspired by optimization meth-ods such as Alternating Direction Method of Multipliers (ADMM) (Taylor et al., 2016; Carreira-Perpinan & Wang, 2014; Choromanska et al., 2018) considers approaches that use auxiliary variables to break optimization into sub-problems.”
Implicit Deep Learning (2019, background): “The idea of lifting the dimension of the training problem in (non-implicit) deep learning by introducing “state” variables has been studied in a variety of works; a non-extensive list includes [50], [5], [24], [56], [57], [12] and [34].”
NAMSG: An Efficient Method For Training Neural Networks (2019, methodology): “AM, AMSGRAD, and RANGER for the tested problems. 1. Introduction and related work Training deep neural networks [6,11,1,10] with large datasets costs a huge amount of time and computational resources [9,33,28,23,31]. In order to accelerate the training process, it is urgently required to improve the”