Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity

By Arnak S. Dalalyan, Alexandre B. Tsybakov. 2008

184 citations recorded by the source corpus.

We study the problem of aggregation under the squared loss in the model of regression with deterministic design. We obtain sharp PAC-Bayesian risk bounds for aggregates defined via exponential weights, under general assumptions on the distribution of errors and on the functions to aggregate. We then apply these results to derive sparsity oracle inequalities.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistics and Probability

In Statistical Learning Theory

Citations in the corpus

Asymptotically minimax empirical Bayes estimation of a sparse normal mean vector (2013, methodology): “The use of pseudo-posteriors, where an inverse temperature parameter plays the role of κ , has been considered in the statistics and machine learning literature (e.g., Dalalyan and Tsybakov 2008; Jiang and Tanner 2008; Zhang 2006), but our context is diﬀerent.”
Exponential Screening and optimal rates of sparse estimation (2010, background): “Dalalyan and Tsybakov (2007, 2008, 2009, 2010) suggested several procedures of this kind based on continuous sparsity priors.”
Approximate Models and Robust Decisions (2014, methodology): “This has the form of a Gibbs posterior or an exponentially weighted PAC-Bayesian approach (Zhang, 2006a, 2006b, Bissiri, Holmes and Walker, 2013, Dalalyan and Tsybakov, 2008, 2012).”
Sparse recovery under matrix uncertainty (2008, methodology): “ue θis reasonably bounded. This oﬀers a possible way of relaxing the strong conditions usually imposed in the context of ℓ1-penalized sparse estimation. Note that another way to do it can be found in [13, 14], suggesting a computationally feasible method of sparse estimation with no assumption on X.”
Bayesian fractional posteriors (2016, methodology): “In contrast, our PAC-Bayes inequality generalizes the results in [16] to a more broader class of models.”
User-friendly introduction to PAC-Bayes bounds (2021, background): “Many aggregates share a formal similarity with the EWA of online learning and with the Gibbs posterior of the PAC-Bayes approach, we refer the reader to [144, 106, 202, 139, 203, 42, 206, 119, 116, 64, 40, 183, 65, 60, 63, 59, 62, 126, 61].”