A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research.

By T. Koo, Ma Li. 2016

25,214 citations recorded by the source corpus.

OBJECTIVE Intraclass correlation coefficient (ICC) is a widely used reliability index in test-retest, intrarater, and interrater reliability analyses. This article introduces the basic concept of ICC in the content of reliability analysis. DISCUSSION FOR RESEARCHERS There are 10 forms of ICCs. Because each form involves distinct assumptions in their calculation and will lead to different interpretations, researchers should explicitly specify the ICC form they used in their calculation. A thorough review of the research design is needed in selecting the appropriate form of ICC to evaluate reliability. The best practice of reporting ICC should include software information, "model," "type," and "definition" selections. DISCUSSION FOR READERS When coming across an article that includes ICC, readers should first check whether information about the ICC form has been reported and if an appropriate ICC form was used. Based on the 95% confident interval of the ICC estimate, values less than 0.5, between 0.5 and 0.75, between 0.75 and 0.9, and greater than 0.90 are indicative of poor, moderate, good, and excellent reliability, respectively. CONCLUSION This article provides a practical guideline for clinical researchers to choose the correct form of ICC and suggests the best practice of reporting ICC parameters in scientific publications. This article also gives readers an appreciation for what to look for when coming across ICC while reading an article.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A computational solution for bolstering reliability of epigenetic clocks: Implications for clinical trials and longitudinal tracking (2021, methodology): “Deviations from this ideal can be quantified using the intraclass correlation coefficient (ICC), a descriptive statistic of the measurement agreement for multiple estimates from the same sample (within-sample variation) relative to other samples (between-sample variation) 13 .”
The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping. (2016, unknown): “Using the lower boundary of the 95% confidence interval of the ICC value (26), reproducibility of each feature was assigned to one of the following categories, as suggested by Koo and Li (27): poor, lower boundary less than 0.”
Applying the self-determination theory (SDT) to explain student engagement in online learning during the COVID-19 pandemic (2021, methodology): “Moreover, the average intraclass correlation coefficients indicated high degrees of reliability of the pre-and post-questionnaires, with values of .97 for perceived autonomy, .97 for perceived competence, and .96 for perceived relatedness (where good > .75, excellent > .90, Koo & Li, 2016).”
Using AI-generated suggestions from ChatGPT to optimize clinical decision support (2023, unknown): “9 represents excellent reliability.(34) Statistical analyses”
A scattered CAT: A critical evaluation of the consensual assessment technique for creativity research. (2019, background): “…(1979) originally identified six different versions of intra-class correlation (ICC), later expanded to ten by McGraw and Wong (1996, as cited in Koo & Li, 2016), which depend on sampling method of judges as well as the items to be rated, single rater reliability vs. mean of multiple raters,…”
Intraclass correlation – A discussion and demonstration of basic features (2019, methodology): “Qualitative interpretations of ICC values are often recommended, for example (under certain restrictions [2]) “poor” (ICC < 0.”
Adapted large language models can outperform medical experts in clinical text summarization (2023, unknown): “We estimated intra-reader correlation based on a mean-rating, fixed-agreement, two-may mixed-effects model 90 using the Pingouin package 91 .”
PD-L1 Immunohistochemistry Comparability Study in Real-Life Clinical Samples: Results of Blueprint Phase 2 Project (2018, background): “9 were considered to indicate good and excellent reliability, respectively.(12) FKS scores of 0.”
Psychological Science Needs a Standard Practice of Reporting the Reliability of Cognitive-Behavioral Measurements (2019, methodology): “Other researchers have suggested that .7 or .8 is a suitable threshold for reliability or have used labels for specific intervals (e.g., .50–.75 = “moderate” reliability, .75–.90 = “good” reliability, and > .90 = “excellent” reliability; Koo & Li, 2016).”
Intracranial EEG signals disentangle multi-areal neural dynamics of vicarious pain perception (2024, background): “We observed excellent between-rater reliabilities (ICC greater than 0.90) 73 for all dimensions of subjective ratings (details in Supplementary Table 5).”
Globally elevated excitation–inhibition ratio in children with autism spectrum disorder and below-average intelligence (2021, background): “000001) [52] and exceeded that in the individual cortical labels, where it ranged from 0.”