Robust parameter estimation with a small bias against heavy contamination

By Hironori Fujisawa, Shinto Eguchi. 2008

308 citations recorded by the source corpus.

In this paper we consider robust parameter estimation based on a certain cross entropy and divergence. The robust estimate is defined as the minimizer of the empirically estimated cross entropy. It is shown that the robust estimate can be regarded as a kind of projection from the viewpoint of a Pythagorean relation based on the divergence. This property implies that the bias caused by outliers can become sufficiently small even in the case of heavy contamination. It is seen that the asymptotic variance of the robust estimator is naturally overweighted in proportion to the ratio of contamination. One may surmise that another form of cross entropy can present the same behavior as that discussed above. It can be proved under some conditions that no cross entropy can present the same behavior except for the cross entropy considered here and its monotone transformation.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/ed9e6b4df60e2ca467044d2bc88a03f89c5f68fe

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation (2021, background): “This divergence has been introduced by (Fujisawa and Eguchi 2008) as a scale-invariant modiﬁca-tion of the robust β -divergences.”
Families of Alpha- Beta- and Gamma- Divergences: Flexible and Robust Measures of Similarities (2010, methodology): “In particular, using the nonlinear transformations (73) for the basic asymmetric Beta-divergence (46), we obtain the Gamma-divergence (72) [35] referred to as here a Beta-Gamma-divergence (D G (P||Q) = D (γ) BG(P||Q))”
Information geometry of divergence functions (2010, methodology): “It is used in machine learning [28] and robust estimation [29–31].”
Mathematical Aspects of Divergence Based Vector Quantization Using Fr´ echet-Derivatives (2009, background): “4 The divergence D (p|| ) is proposed to be robust for 2 [0,1] with existence of D =0 in the limit ! 0. A detailed analysis of robustness is given in [ 17 ].”
Density-ratio matching under the Bregman divergence: a unified framework of density-ratio estimation (2012, background): “Although there seems to be no universal way for this (Basu et al. 1998; Jones et al. 2001; Fujisawa and Eguchi 2008), a practical approach would be to use crossvalidation over a fixed divergence such as the squared distance.”