Do multiple outcome measures require p-value adjustment?

By R. Feise. BMC Medical Research Methodology, 2002

1,351 citations recorded by the source corpus.

BackgroundReaders may question the interpretation of findings in clinical trials when multiple outcome measures are used without adjustment of the p-value. This question arises because of the increased risk of Type I errors (findings of false "significance") when multiple simultaneous hypotheses are tested at set p-values. The primary aim of this study was to estimate the need to make appropriate p-value adjustments in clinical trials to compensate for a possible increased risk in committing Type I errors when multiple outcome measures are used.DiscussionThe classicists believe that the chance of finding at least one test statistically significant due to chance and incorrectly declaring a difference increases as the number of comparisons increases. The rationalists have the following objections to that theory: 1) P-value adjustments are calculated based on how many tests are to be considered, and that number has been defined arbitrarily and variably; 2) P-value adjustments reduce the chance of making type I errors, but they increase the chance of making type II errors or needing to increase the sample size.SummaryReaders should balance a study's statistical significance with the magnitude of effect, the quality of the study and with findings from other studies. Researchers facing multiple outcome measures might want to either select a primary outcome measure or use a global assessment measure, rather than adjusting the p-value.

Subfields: Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/8f45f2549df91a55f460989b1fd24c8c4a6a3078

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

When to adjust alpha during multiple testing: a consideration of disjunction, conjunction, and individual testing (2021, background): “…a single significant result that is obtained following 20 individual tests that each use an αIndividual of 0.05 is more likely to be a Type I error than a single significant result that is obtained using a single individual test that uses an αIndividual of 0.05 (e.g., Feise, 2002; Sainani, 2009).”
Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate (2017, background): “The first problem is that it leads to confusion over which hypotheses to include in the family of hypotheses that is used to compute an adjusted alpha level (Feise, 2002; O’Keefe, 2003, 2007; Trafimow & Earp, 2017).”
Youth Empowerment Programs for Improving Adolescents’ Self-Efficacy and Self-Esteem (2013, background): “Given the large number of outcomes measured by each study, however, some of these significant outcomes are likely to have occurred by chance alone (Feise, 2002).”
Remote sensing techniques for mangrove mapping (2006, background): “One of the major reasons was the increasing chance of the TYPE I error that usually happened when conducting multiple hypothesis testing (Hsu, 1996; Rothman, 1990; Perneger, 1998; Feise, 2002).”
An introduction to multiplicity issues in clinical trials: the what, why, when and how. (2016, neutral): “If we test each of the individual outcomes separately at a nominal 5% level and obtain any significant difference, the probability of a spurious claim of significance is higher than the anticipated 5%.(2,16,17) To avoid inflation of the type I error rate, several solutions have been proposed.”
Adjust for Multiple Comparisons? It's Not That Simple. (2016, background): “the tests performed in large randomized controlled trials with planned substudies be required to meet a different significance threshold than a small laboratory-based study with only one or two comparisons of interest? (3) If one stipulates that the researcher need not consider the above, but still ”
Don't fear ‘fear conditioning’: Methodological considerations for the design and analysis of studies on human fear acquisition, extinction, and return of fear (2017, neutral): “In case such different predictions exist with respect to multiple indices of conditioned responding or if one outcome measure serves as the primary measure, a correction for multiple comparisons is generally not necessary (Feise, 2002).”
Escaping the Bonferroni iron claw in ecological studies (2004, background): “In other contexts, even stronger claims against alpha corrections have recently been the subject of controversy (Perneger 1998, 1999, Feise 2002).”
Strengthening statistical usage in marine ecology (2012, background): “Correction only under certain circumstances (which are rather rare inmarine biology/ecology)has also been advocated (Cook and Farewell, 1996), as has been downplaying the importance of correction, in favor of effect size, study design, and prior judgment-determined outcome measures (Feise, 2002).”