Distributed optimization of deeply nested systems

By M. A. Carreira-Perpiñán, Weiran Wang. 2012

213 citations recorded by the source corpus.

In science and engineering, intelligent processing of complex signals such as images, sound or language is often performed by a parameterized hierarchy of nonlinear processing layers, sometimes biologically inspired. Hierarchical systems (or, more generally, nested systems) offer a way to generate complex mappings using simple stages. Each layer performs a different operation and achieves an ever more sophisticated representation of the input, as, for example, in an deep artificial neural network, an object recognition cascade in computer vision or a speech front-end processing. Joint estimation of the parameters of all the layers and selection of an optimal architecture is widely considered to be a difficult numerical nonconvex optimization problem, difficult to parallelize for execution in a distributed computation environment, and requiring significant human expert effort, which leads to suboptimal systems in practice. We describe a general mathematical strategy to learn the parameters and, to some extent, the architecture of nested systems, called the method of auxiliary coordinates (MAC). This replaces the original problem involving a deeply nested function with a constrained problem involving a different function in an augmented space without nesting. The constrained problem may be solved with penalty-based methods using alternating optimization over the parameters and the auxiliary coordinates. MAC has provable convergence, is easy to implement reusing existing algorithms for single layers, can be parallelized trivially and massively, applies even when parameter derivatives are not available or not desirable, and is competitive with state-of-the-art nonlinear optimizers even in the serial computation setting, often providing reasonable models within a few iterations.

Subfields: Artificial Intelligence, Computational Mechanics, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Optimization & Training

In Signal Processing

Citations in the corpus

Deeply-Supervised Nets (2014, methodology): “Finally, we mention [4]: this work reexpresses the original output layer classification problem in an equivalent decouple-able form.”
Training Neural Networks Using Features Replay (2018, methodology): “[4] and [34] avoid the backward locking by removing the backpropagation algorithm completely.”
Decoupled Neural Interfaces using Synthetic Gradients (2016, neutral): “…Real-Time Recurrent Learning (Williams & Zipser, 1989) or approximations such as (Ollivier & Charpiat, 2015; Tallec & Ollivier, 2017) may seem a promising way to remove update locking, these methods require maintaining the full (or approximate) gradient of the current state with respect to the…”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, methodology): “Although the approach to generate the binary AE is diﬀerent from the previous one, since they achieve hashing with binary AEs helped by MAC ( Method of Auxiliary Coordinates ) [134], the proposal in [135] is quite similar.”
How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation (2014, background): “This target propagation viewpoint is also related to optimization approaches where the representation values (i.e., hidden unit activations) are free variables that can be optimized over (Carreira-Perpinan and Wang, 2014).”
Theory of cortical function (2017, background): “Normalization in mammalian cortex, however, does not rely on GABA inhibition (70), but rather is caused by a decrease in excitation (71).”
Training Neural Networks Without Gradients: A Scalable ADMM Approach (2016, methodology): “Note that our work is the ﬁrst to consider alternating least squares as a method to distribute computation across a cluster, although the authors of (Carreira-Perpin´an & Wang, 2012) do consider implementations that are “distributed” in the sense of using multiple threads on a single machine via…”
Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer (2020, methodology): “The intuition is very similar to the “model selection on the fly” idea of [7].”
Implicit Deep Learning (2019, methodology): “he dimension of the training problem in (non-implicit) deep learning by introducing “state” variables has been studied in a variety of works; a non-extensive list includes [13], [2], [7], [17], [18], [5] and [12]. Lifted models are trained using block coordinate descent methods, Alternating Directio”
Decoupled Parallel Backpropagation with Convergence Guarantee (2018, background): “In [4, 23], the authors loosened the exact connections between layers by introducing auxiliary variables.”
Training Deep Networks with Structured Layers by Matrix Backpropagation (2015, background): “deep architectures using stochastic gradient descent, based on very large datasets, has produced impressive results in realistic settings, for a variety of computer vision and machine learning domains[2, 3,4,5]. There is now a renewed enthusiasm of creating integrated, automatic models that can hand”
Decoupled Greedy Learning of CNNs (2019, methodology): “Another line of related work inspired by optimization methods such as Alternating Direction Method of Multipliers (ADMM) [29, 5, 6] use auxiliary variables to break the optimization into sub-problems.”
Difference Target Propagation (2014, background): “A similar idea was recently proposed where the constraints are relaxed into penalties, yielding a different (iterative) way to optimize deep networks [9].”
Structured Pruning of Neural Networks With Budget-Aware Regularization (2018, methodology): “Another budgeted pruning approach is “LearningCompression” [4], which uses the method of auxiliary coordinates [3] instead of back-propagation.”