An equivalence between high dimensional Bayes optimal inference and M-estimation
By Madhu S. Advani, Surya Ganguli. 2016
16 citations recorded by the source corpus.
When recovering an unknown signal from noisy measurements, the computational difficulty of performing optimal Bayesian MMSE (minimum mean squared error) inference often necessitates the use of maximum a posteriori (MAP) inference, a special case of regularized M-estimation, as a surrogate. However, MAP is suboptimal in high dimensions, when the number of unknown signal components is similar to the number of measurements. In this work we demonstrate, when the signal distribution and the likelihood function associated with the noise are both log-concave, that optimal MMSE performance is asymptotically achievable via another M-estimation procedure. This procedure involves minimizing convex loss and regularizer functions that are nonlinearly smoothed versions of the widely applied MAP optimization problem. Our findings provide a new heuristic derivation and interpretation for recent optimal M-estimators found in the setting of linear measurements and additive noise, and further extend these results to nonlinear measurements with non-additive noise. We numerically demonstrate superior performance of our optimal M-estimators relative to MAP. Overall, at the heart of our work is the revelation of a remarkable equivalence between two seemingly very different computational problems: namely that of high dimensional Bayesian integration underlying MMSE inference, and high dimensional convex optimization underlying M-estimation. In essence we show that the former difficult integral may be computed by solving the latter, simpler optimization problem.
Subfields: Computational Mechanics, Computer Networks and Communications, Statistics and Probability, Probabilistic & Bayesian Methods, Statistical Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
In Computer Networks and Communications
In Statistics and Probability
In Probabilistic & Bayesian Methods
In Statistical Learning Theory
Citations in the corpus
- Optimal errors and phase transitions in high-dimensional generalized linear models (2017, background): “Random instances were studied in particular in the context of so-called M estimators (18–21). iii ) In studies of artificial neural networks there has been a large amount of work using random instances of GLMs, with ϕ playing the role of a nonlinear activation function.”