Safe Testing

By Peter D. Grünwald, Rianne de Heide, Wouter M. Koolen. Information Theory and Applications Workshop, 2019

332 citations recorded by the source corpus.

We present a new theory of hypothesis testing. The main concept is the s-value, a notion of evidence which, unlike p-values, allows for effortlessly combining evidence from several tests, even in the common scenario where the decision to perform a new test depends on the previous test outcome: safe tests based on s-values generally preserve Type-I error guarantees under such ‘optional continuation’. S-values exist for completely general testing problems with composite null and alternatives. Their prime interpretation is in terms of gambling or investing, each S-value corresponding to a particular investment. Surprisingly, optimal "GROW" S-values, which lead to fastest capital growth, are fully characterized by the joint information projection (JIPr) between the set of all Bayes marginal distributions on ${\mathcal{H}_0}$ and ${\mathcal{H}_1}$. Thus, optimal s-values also have an interpretation as Bayes factors, with priors given by the JIPr. We illustrate the theory using two classical testing scenarios: the one-sample t-test and the 2 × 2-contingency table. In the t-test setting, GROW S-values correspond to adopting the right Haar prior on the variance, like in Jeffreys’ Bayesian t-test. However, unlike Jeffreys’, the default safe t-test puts a discrete 2-point prior on the effect size, leading to better behaviour in terms of statistical power. Sharing Fisherian, Neymanian and Jeffreys-Bayesian interpretations, S-values and safe tests may provide a methodology acceptable to adherents of all three schools.

Subfields: Artificial Intelligence, Economics and Econometrics, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/f5f628da62f0d4647a9ae51a18723c924b7498b5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Statistical and Nonlinear Physics

Citations in the corpus

False discovery rate control with e‐values (2020, methodology): “We use ‘e-value’ as an abstract umbrella term which encompasses betting scores, likelihood ratios, and stopped supermartingales, which appear in the recent literature, for example, Shafer (2021), Grünwald et al. (2020), Wasserman et al. (2020) and Howard et al. (2020, 2021).”
Admissible ways of merging p-values under arbitrary dependence (2020, positive): “Second, we give two representation results (Theorems 5.1 and 5.2) of admissible p-merging functions which are naturally connected to e-values (Vovk and Wang, 2020b; Shafer, 2019; Gr¨unwald et al., 2020), our important technical tool, via a duality argument.”
E-values: Calibration, combination and applications (2019, neutral): “[2020b], and Grünwald et al. [2020], meaning a nonnegative supermartingale with initial value 1).”
Sample Size Justification (2021, neutral): “…Sequential Probability Ratio Test (Wald, 1945), combining independent statistical tests (Westberg, 1985), group sequential designs (Jennison & Turnbull, 2000), sequential Bayes factors (Schön-brodt, Wagenmakers, Zehetleitner, & Perugini, 2017), and safe testing (Grünwald, de Heide, & Koolen, 2019).”
Estimating means of bounded random variables by betting (2020, background): “In particular, Shafer espouses complementary benefits of such approaches, ranging from improved scientific communication, ties to historical advances in probability, and reproducibility via continued experimentation (also see [25]), but our focus here has been on developing a new state of the art fo”