Tweedie’s Formula and Selection Bias

By Bradley Efron. 2011

957 citations recorded by the source corpus.

We suppose that the statistician observes some large number of estimates zi, each with its own unobserved expectation parameter μi. The largest few of the zi’s are likely to substantially overestimate their corresponding μi’s, this being an example of selection bias, or regression to the mean. Tweedie’s formula, first reported by Robbins in 1956, offers a simple empirical Bayes approach for correcting selection bias. This article investigates its merits and limitations. In addition to the methodology, Tweedie’s formula raises more general questions concerning empirical Bayes theory, discussed here as “relevance” and “empirical Bayes information.” There is a close connection between applications of the formula and James–Stein estimation.

Subfields: Artificial Intelligence, Statistics and Probability, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Optimization & Training

Citations in the corpus

Improved Techniques for Training Score-Based Generative Models (2020, methodology): “Instead of directly returning xT , this denoising step returns xT + σ 2 T sθ(xT , σT ) (see Algorithm 1), which essentially removes the unwanted noise N (0, σ(2) T I) from xT using Tweedie’s formula [14].”
Score-Based Generative Modeling through Stochastic Differential Equations (2020, positive): “In all experiments of this paper we ensure there is a single denoising step at the end of sampling, using Tweedie’s formula (Efron, 2011). denote the corresponding interpolated score-based model at 2 N time steps.”
Gotta Go Fast When Generating Data with Score-Based Models (2021, background): “This corresponds to: At the last iteration, this incorrect denoising would be: for VE and Meanwhile, the correct way to denoise based on Tweedie formula [Efron, 2011] is: where Var [ x ( t ) | x (0)] is the variance of the transition kernel: Var [ x ( t ) | x (0)] = σ min = 0 .”
Efficient Policy Learning (2017, methodology): “Meanwhile, we fit the conditional density function f(· ∣∣Xi) by adapting Lindsey’s method, a technique for estimating distribution functions using software for generalized linear modeling (Efron and Tibshirani, 1996; Lindsey, 1974). In the case without covariates, Lindsey’s method involves first dis”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, background): “A notable property of the score function ∇ x t log p t ( x t ) is that it re-covers the minimum mean squared error (MMSE) estimator of x given x t [76, 16, 68]: where we can essentially treat ˜ x as a “denoised” version of x t .”
Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (2023, methodology): “In particular, knowing all p t ( y ) p t ( x ) enables optimal denoising, analogous to Tweedie’s theorem (Efron, 2011): Theorem 4.1 (Discrete Tweedie’s Theorem) .”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, neutral): “√ √ Let C be the conditional embeddings, and the perturbed data Since ϵ ξ ( z t ; t ) = − σ t s ξ ( z t ; t ) as derived from Tweedie’s (Efron, 2011; Luo, 2022), it is equivalent to denoising score matching of ∇ z t log p ξ ( z t ) (Hyvärinen, 2005; Vincent, 2011), thus DMs are also called…”
SLOPE—ADAPTIVE VARIABLE SELECTION VIA CONVEX OPTIMIZATION (2014, background): “…Roeder (2009); Meinshausen et al. (2009); Berk et al. (2013); Zhang and Zhang (2013); van de Geer et al. (2014); Meinshausen and Bühlmann (2010); Efron (2011); Javanmard 4Observe that when all the λi’s take on an identical positive value, the sorted `1 norm reduces to the usual `1 norm (up to…”
Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion (2023, background): “Here, E p t 0 ( x | x t ) [ x x t ] = x t + t ∇ log p t ( x t ) is the denoiser function (Efron, 2011), an alternative expression for the score function ∇ log p t ( x t ) .”
Forecasting With Dynamic Panel Data Models (2017, positive): “Recently, it was extended by Efron (2011) to the family of exponential distribution, allowing for a unknown finite-dimensional parameter θ. Theorem 4.2 extends Tweedie’s formula to the estimation of correlated random effect parameters in a dynamic panel regression setup.”
Understanding Diffusion Models: A Unified Perspective (2022, methodology): “Mathematically, for a Gaussian variable z ∼ N (z;µz,Σz), Tweedie’s Formula states that: E [µz|z] = z +Σz∇z log p(z) In this case, we apply it to predict the true posterior mean of xt given its samples.”
Diffusion Posterior Sampling for General Noisy Inverse Problems (2022, methodology): “A classic result of Tweedie’s formula (Robbins, 1992; Stein, 1981; Efron, 2011; Kim & Ye, 2021) states that one can retrieve the empirical Bayes optimal posterior mean ˆ x 0 using the formula in (10).”
Empirical Bayesball Remixed: Empirical Bayes Methods for Longitudinal Data (2017, methodology): “To compare the performance of the foregoing predictions with more classical Bayesian methods we have also considered an alternative nonparametric formulation of the binomial model employing a Dirichlet process prior as introduced by Ferguson (1973), Antoniak (1974) and Ferguson (1983).”
Bayesian weighted Mendelian randomization for causal inference based on summary statistics (2018, background): “In fact, this phenomenon should be attributed to winner’s curse in the context of GWAS (Zhong and Prentice, 2008) or selection bias in statistical literature (Efron, 2011).”
Optimal Inference After Model Selection (2014, background): “An empirical Bayes approach for selection-adjusted estimation can be found in Efron (2011).”
Diffusion Normalizing Flow (2021, methodology): “[19], adding noise at the last step will significantly lower sampling quality, we use one single denoising step at the end of sampling with Tweedie’s formula [10].”
Policy Learning With Observational Data (2017, background): “Recently, Armstrong and Koles´ar (2017) showed that, in a class of average treatment eﬀect estimation problems, ﬁnite sample conditionally minimax linear estimators are asymptotically eﬃcient, thus providing a connection between desirable ﬁnite sample guarantees and asymptotic optimality.”