Least Squares Estimation Without Priors or Supervision

By Martin Raphan, Eero P. Simoncelli. 2011

109 citations recorded by the source corpus.

Subfields: Statistics and Probability, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

In Optimization & Training

Citations in the corpus

Improved Techniques for Training Score-Based Generative Models (2020, methodology): “First, the score function is learned via denoising score matching [3, 4, 5].”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “The method of denoising score matching can also be derived from the perspective of Bayes least squares [43, 44], using techniques of Stein’s Unbiased Risk Estimator [35, 56].”
Image Super-Resolution via Iterative Refinement (2021, methodology): “Our approach is also linked to denoising score matching [35], [36], [37], [38] for training unnormalized energy”
Denoising Diffusion Implicit Models (2020, methodology): “DDPMs optimize a variational lower bound to the log-likelihood, whereas NCSNs optimize the score matching objective (Hyv ¨ arinen, 2005) over a nonparametric Parzen density estimator of the data (Vincent, 2011; Raphan & Simoncelli, 2011).”
Density Modeling of Images using a Generalized Normalization Transformation (2015, neutral): “Note that, counterintuitively, this expression does not refer directly to the prior density, but it is nevertheless exactly equivalent to the Bayesian least squares solution (Raphan & Simoncelli, 2011).”
Adversarial score matching and improved sampling for image generation (2020, methodology): “The main beneﬁts of their approach (DSM-ALS) are that it produces relatively high-quality samples and guarantees high diversity due to ALS.”
Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser (2020, background): “This result is related to score matching, but provides more direct relationship between least-squares optimal denoising and the embedded prior [27].”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “, score estimate) has established techniques such as score matching [101], denoising score matching [188, 189, 238], and sliced score matching [222], so we can directly employ one of them to train our noise-conditional score networks from perturbed data points.”
On Maximum Likelihood Training of Score-Based Generative Models (2021, methodology): “(1) so that samples from p t p x q are available in closed form, and using techniques like Denoising Score Matching [Vincent, 2011; Raphan and Simoncelli, 2007, 2011] or Sliced Score Matching [Song et al., 2019] to sidestep the need for explicit ground-truth scores.”
Stochastic Solutions for Linear Inverse Problems using the Prior Implicit in a Denoiser (2021, methodology): “The idea was introduced in [46], extended to the case of Gaussian additive noise in [9] (see also [12]), and generalized to many other measurement models [11].”
How to Train Your Energy-Based Models (2021, background): “Similar formulations are also explored by Raphan and Simoncelli (2007, 2011) and can be traced back to Tweedie’s formula (1956) and Stein’s Unbiased Risk Estimation (Stein, 1981).”
Deep Mean-Shift Priors for Image Restoration (2017, methodology): “Note that this is the same loss as in non-parametric least squares estimators [23, 26, 20].”
Stein Unbiased GrAdient estimator of the Risk (SUGAR) for Multiple Parameter Selection (2014, methodology): “Toward this goal, we adopt the framework of the (generalized) Stein Unbiased Risk Estimator (SURE) [28, 48, 55, 62, 66].”
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “The training objective for s θ ( x , t ) is a mixture of de-noising score matching losses [58, 44, 43] over randomly sampled time steps.”
Neural Empirical Bayes (2019, background): “However, this equivalence breaks down beyond gaussian kernels—see (Raphan and Simoncelli, 2011) for a comprehensive survey of empirical Bayes least squares estimators.”
Energy-Based Imitation Learning (2020, methodology): “Then we can use the DAE ĝ to approximate the score function ∇y log p(y) of the corrupted distribution by ∇y log p(y) ∝ ĝ(y) − y [35, 27, 33].”
Improved Autoregressive Modeling with Distribution Smoothing (2021, background): “(2) only works when q (˜ x | x ) is Gaussian (though alternative denoising updates for other smoothing processes could be derived under the Empirical Bayes framework (Raphan & Simoncelli, 2011)).”