Neural Taylor Approximations: Convergence and Exploration in Rectifier Networks

By David Balduzzi, Brian McWilliams, Tony Butler-Yeoman. 2016

31 citations recorded by the source corpus.

Modern convolutional networks, incorporating rectifiers and max-pooling, are neither smooth nor convex; standard guarantees therefore do not apply. Nevertheless, methods from convex optimization such as gradient descent and Adam are widely used as building blocks for deep learning algorithms. This paper provides the first convergence guarantee applicable to modern convnets, which furthermore matches a lower bound for convex nonsmooth functions. The key technical tool is the neural Taylor approximation -- a straightforward application of Taylor expansions to neural networks -- and the associated Taylor loss. Experiments on a range of optimizers, layers, and tasks provide evidence that the analysis accurately captures the dynamics of neural optimization. The second half of the paper applies the Taylor approximation to isolate the main difficulty in training rectifier nets -- that gradients are shattered -- and investigates the hypothesis that, by exploring the space of activation configurations more thoroughly, adaptive optimizers such as RMSProp and Adam are able to converge to better solutions.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Adversarial Vulnerability of Neural Networks Increases With Input Dimension (2018, background): “If we follow common practice and initialize our nets as proposed by He et al. (2015), then H3-H5 are satisﬁed at initialization by design, while H1 is usually a very good approximation (Balduzzi et al., 2017).”
First-Order Adversarial Vulnerability of Neural Networks and Input Dimension (2018, background): “(2015), then H3-H5 are satisfied at initialization by design, while H1 is usually a very good approximation (Balduzzi et al., 2017).”
Kernel-predicting convolutional networks for denoising Monte Carlo renderings (2017, background): “Despite their C1 discontinuity, ReLUs have been shown to achieve state-of-the-art performance in many tasks and are known to encourage the (non-convex) optimization procedure to nd better local minima [Balduzzi et al. 2016].”