Stein Unbiased GrAdient estimator of the Risk (SUGAR) for Multiple Parameter Selection

By Charles-Alban Deledalle, Samuel Vaiter, Mohamed-Jalal Fadili, G. Peyr'e. 2014

125 citations recorded by the source corpus.

Algorithms to solve variational regularization of ill-posed inverse problems usually involve operators that depend on a collection of continuous parameters. When these operators enjoy some (local) regularity, these parameters can be selected using the so-called Stein Unbiased Risk Estimate (SURE). While this selection is usually performed by exhaustive search, we address in this work the problem of using the SURE to efficiently optimize for a collection of continuous parameters of the model. When considering non-smooth regularizers, such as the popular l1-norm corresponding to soft-thresholding mapping, the SURE is a discontinuous function of the parameters preventing the use of gradient descent optimization techniques. Instead, we focus on an approximation of the SURE based on finite differences as proposed in (Ramani et al., 2008). Under mild assumptions on the estimation mapping, we show that this approximation is a weakly differentiable function of the parameters and its weak gradient, coined the Stein Unbiased GrAdient estimator of the Risk (SUGAR), provides an asymptotically (with respect to the data dimension) unbiased estimate of the gradient of the risk. Moreover, in the particular case of soft-thresholding, the SUGAR is proved to be also a consistent estimator. The SUGAR can then be used as a basis to perform a quasi-Newton optimization. The computation of the SUGAR relies on the closed-form (weak) differentiation of the non-smooth function. We provide its expression for a large class of iterative proximal splitting methods and apply our strategy to regularizations involving non-smooth convex structured penalties. Illustrations on various image restoration and matrix completion problems are given.

Subfields: Computational Mechanics, Mathematical Physics, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Mathematical Physics

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Hyperparameter optimization with approximate gradient (2016, background): “This is because Grid Search and SMBO naturally start from a pre-deﬁned grid that starts from the extremes of the interval, while random search simply chooses a random point from the domain.”
Efficient and Modular Implicit Differentiation (2021, methodology): “The first one consists in unrolling the iterations of an optimization algorithm and to use the final iteration as a proxy for the optimization problem solution [68, 28, 25, 31].”
Bilevel Optimization with Nonsmooth Lower Level Problems (2015, background): “In [21], in the setting of unbiased risk estimation and parameter selection, iterative (weak) differentiation of Euclidean proximal splitting algorithms is studied.”
Techniques for Gradient-Based Bilevel Optimization with Non-smooth Lower Level Problems (2016, methodology): “The approach in [11] also considers an algorithm replacing the non-smooth lower level problem.”