Empirical risk minimization for heavy-tailed losses

By Christian T. Brownlees, Émilien Joly, Gábor Lugosi. 2014

154 citations recorded by the source corpus.

The purpose of this paper is to discuss empirical risk minimization when the losses are not necessarily bounded and may have a distribution with heavy tails. In such situations, usual empirical averages may fail to provide reliable estimates and empirical risk minimization may provide large excess risk. However, some robust mean estimators proposed in the literature may be used to replace empirical means. In this paper, we investigate empirical risk minimization based on a robust estimate proposed by Catoni. We develop performance bounds based on chaining arguments tailored to Catoni's mean estimator.

Subfields: Finance, Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/aa1f3de39d87c9dfb778335ea3e89ff05a000099

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Finance

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Sub-Gaussian mean estimators (2015, unknown): “It is shown in Bubeck, Cesa-Bianchi, and Lugosi [4] that for each M > 0, α ∈ (0 , 1], and δ , there exists an estimator (cid:98) E n ( X n 1 , δ ) such that sup The estimator (cid:98) E n ( X n 1 , δ ) satisfying this bound is the median-of-means estimator with appropriately chosen parameters.”
Risk minimization by median-of-means tournaments (2016, unknown): “We mention here that Brownlees, Joly, and Lugosi [8] propose empirical minimization based on a diﬀerent robust mean estimator, a carefully designed M-estimator proposed by Catoni [11].”
VALUATION : A N ADVERSARIAL APPROACH TO UNCOVER CATASTROPHIC FAILURES (2018, unknown): “While the exact bound on the error in the empirical estimate may depend on the particular estimator and concentration inequalities being used (Brownlees et al., 2015; Audibert et al., 2011), any approach relying solely on uninformed random sampling will face similar issues.”
Convergence rates of least squares regression estimators with heavy-tailed errors (2017, background): “t on the errors, we refer the reader to [7]. It is worthwhile to note that the shortcomings of the LSE quantiﬁed here also rigorously justify the motivation of developing other robust procedures (cf. [4, 14, 15, 18, 19, 24, 35, 36, 45, 46, 47, 57]). Remark 9. Our Theorems 3 and 4 show that the momen”
Adaptive Huber Regression (2017, background): “Beyond mean estimation, Brownlees, Joly, and Lugosi (2015) extended Catoni’s idea to study empirical risk minimization problems when the losses are unbounded.”
Sub-Gaussian estimators of the mean of a random matrix with heavy-tailed entries (2016, background): “eakdown point, we refer the reader to [22]. S. Minsker/ Estimators of the mean of a random matrix 3 the data. Ideas behind these results have also been extended to empirical risk minimization methods [6,32] which cover a wide range of statistical applications. Let us emphasize that the aforementione”
Robust machine learning by median-of-means: Theory and practice (2017, background): “…subgaussian concentration bounds [25], the other estimators are either suboptimal see for example [71] for the estimation of the mean of a vector in R d or [18] for more general learning frameworks, or completely untractable see for example [61, 59, 50] for some examples in regression frameworks.”
Adversarial Risk and the Dangers of Evaluating Against Weak Attacks (2018, background): “Second, if the loss function is unbounded, the variance of the estimator may be arbitrarily large (Brownlees et al., 2015; Audibert et al., 2011).”
Regularization, sparse recovery, and median-of-means tournaments (2017, methodology): “We refer to Audibert and Catoni [1], Hsu and Sabato [4], Lerasle and Oliveira [9], Minsker [17], Brownlees, Joly, and Lugosi [3], Lugosi and Mendelson [10] for a sample of the literature.”
Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations (2015, background): “Indeed, under a heavy-tailed distribution the sample average of the loss corresponding to any ﬁxed decision x ∈ X may not even converge to the expected loss; see e.g. [13, 15].”