Super-efficiency of automatic differentiation for functions defined as a minimum

By Pierre Ablin, G. Peyr'e, Thomas Moreau. 2020

42 citations recorded by the source corpus.

In min-min optimization or max-min optimization, one has to compute the gradient of a function defined as a minimum. In most cases, the minimum has no closed-form, and an approximation is obtained via an iterative algorithm. There are two usual ways of estimating the gradient of the function: using either an analytic formula obtained by assuming exactness of the approximation, or automatic differentiation through the algorithm. In this paper, we study the asymptotic error made by these estimators as a function of the optimization error. We find that the error of the automatic estimator is close to the square of the error of the analytic estimator, reflecting a super-efficiency phenomenon. The convergence of the automatic estimator greatly depends on the convergence of the Jacobian of the algorithm. We analyze it for gradient descent and stochastic gradient descent and derive convergence rates for the estimators in these cases. Our analysis is backed by numerical experiments on toy problems and on Wasserstein barycenter computation. Finally, we discuss the computational complexity of these estimators and give practical guidelines to chose between them.

Subfields: Artificial Intelligence, Computational Mechanics, Management Science and Operations Research, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Management Science and Operations Research

In Optimization & Training

Citations in the corpus

Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “This observation is inline with theoretical ﬁndings on strongly convex inner objectives (Blondel et al., 2021; Ablin et al., 2020), which indicate that implicit differentiation is better at approximating the Jacobian when the optimizer solution is inexact.”
Gradients are Not All You Need (2021, background): “Automatic differentiation provides a conceptually straightforward handle for computing derivatives though these systems, and often can be applied with limited compute and memory overhead [Paszke et al., 2017, Ablin et al., 2020, Margossian, 2019, Bischof et al., 1991, Corliss et al., 2013].”
Efficient and Modular Implicit Differentiation (2021, methodology): “The first one consists of unrolling the iterations of an optimization algorithm and using the final iteration as a proxy for the optimization problem solution [76, 30, 27, 37, 1].”