pROC: an open-source package for R and S+ to analyze and compare ROC curves
By Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, F. Lisacek, Jean-Charles Sanchez, Markus Müller. 2011
12,241 citations recorded by the source corpus.
BackgroundReceiver operating characteristic (ROC) curves are useful tools to evaluate classifiers in biomedical and bioinformatics applications. However, conclusions are often reached through inconsistent use or insufficient statistical analysis. To support researchers in their ROC curves analysis we developed pROC, a package for R and S+ that contains a set of tools displaying, analyzing, smoothing and comparing ROC curves in a user-friendly, object-oriented and flexible interface.ResultsWith data previously imported into the R or S+ environment, the pROC package builds ROC curves and includes functions for computing confidence intervals, statistical tests for comparing total or partial area under the curve or the operating points of different classifiers, and methods for smoothing ROC curves. Intermediary and final results are visualised in user-friendly interfaces. A case study based on published clinical and biomarker data shows how to perform a typical ROC analysis with pROC.ConclusionspROC is a package for R and S+ specifically dedicated to ROC analysis. It proposes multiple statistical tests to compare ROC curves, and in particular partial areas under the curve, allowing proper ROC interpretation. pROC is available in two versions: in the R programming language or with a graphical user interface in the S+ statistical software. It is accessible at http://expasy.org/tools/pROC/ under the GNU General Public License. It is also distributed through the CRAN and CSAN public repositories, facilitating its installation.
Subfields: Artificial Intelligence, Epidemiology, Statistics and Probability, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- F. Nahm
- Noel C. F. Codella
- Kjell Johnson
- Iain James Marshall
- Byron C. Wallace
- Philippe Lambin
- Christopher S. McGinnis
- Lyndsay M. Murrow
In Epidemiology
- P. Meredith
- Gary B. Smith
- Noel C. F. Codella
- D. Prytherch
- Ilker Ünal
- Joel T. Dudley
- Jalal Al-Tamimi
- Lionel Tarassenko
In Statistics and Probability
- F. Nahm
- Gerrit Hirschfeld
- Christian Thiele
- Antoine Guisan
- A. Bergamini
- F. Breiner
- M. Nobis
- Selçuk Korkmaz
In Evaluation & Benchmarks
- Ilker Ünal
- Anna Noel-Storr
- Jalal Al-Tamimi
- Gerrit Hirschfeld
- Anirban Mukhopadhyay
- Marco A. F. Pimentel
- A. Bergamini
- David T. Jones
Citations in the corpus
- Deep learning predicts hip fracture using confounding patient and healthcare variables (2018, methodology): “Classifier Receiver Operating Curves All Receiver Operator Curves (ROC) analyses and comparisons were done with the R pROC package.(45) We compute Area Under the Curve (AUC) with 95% confidence intervals determined by 2000 bootstrapped replicates stratified to control case-control frequency.”
- Distributed learning: Developing a predictive model based on data from multiple hospitals without data leaving the hospital - A real life proof of concept. (2016, methodology): “3 (R foundation for statistical computing, Vienna, Austria) using the pROC [37] and cvAUC [38] packages.”
- Species distribution modelling for the people: unclassified landsat TM imagery predicts bird occurrence at fine resolutions (2013, methodology): “We evaluated prediction performance of the models using the area under the receiver operating characteristic curve (AUC) on the independent test set, for which we computed 95% DeLong confidence intervals using the ‘pROC’ R package (Robin et al., 2011).”
- Structural Remodeling of the Human Colonic Mesenchyme in Inflammatory Bowel Disease (2018, methodology): “Model classification performance was assessed using ‘pROC’ R package (Robin et al., 2011) by computing Area Under Receiver Operating Curve (AUC) and examination of confusion matrices.”
- Word2Vec inversion and traditional text classifiers for phenotyping lupus (2017, methodology): “2 External AUC Curves This graph depicts the AUC of each technique as it performed on the external testing set, generated using the pROC package [42] with BOWs, some of the top words for classification were capturing writing styles by SLE researchers, as well as identification numbers for these…”
- Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classification: an open, web-based, international, diagnostic study (2019, methodology): “Receiver operating characteristic curves, areas under the curves, and their 95% CIs were calculated with pROC, 19 and we compared areas under the curves with the method described by Delong and colleagues.”
- Comparison of Machine Learning Classifiers for Influenza Detection from Emergency Department Free-text Reports (2015, methodology): “[43], which is a commonly used method by biomedical researchers using the R package pROC [44].”
- easyROC: An Interactive Web-tool for ROC Curve Analysis Using R Language Environment (2016, methodology): “The R environment includes comprehensive tools for ROC curve analysis, such as ROCR (Sing et al., 2005), pROC (Robin et al., 2011), ROC (Carey and Redestig, 2015) and OptimalCutpoints (Lopez-Raton et al., 2014).”
- DISOPRED3: precise disordered region predictions with annotated protein-binding activity (2014, methodology): “Finally, ROC curves were generated from the pairs (1-specificity, sensitivity) corresponding to decreasing values of v and the area under the curve (AUC) was calculated with the trapezoid integration method in the pROC package (Robin et al. , 2011) for R (R Core Team, 2012).”
- Tumor aneuploidy correlates with markers of immune evasion and with reduced response to immunotherapy (2017, methodology): “This model built on the training set was then applied to the test set, and the ROC-AUC (area under the curve of the receiveroperating characteristic) was calculated to assess the sensitivity and specificity of the model itself [using R package pROC (68)].”
- Feed-forward Hierarchical Model of the Ventral Visual Stream Applied to Functional Brain Image Classification (2012, neutral): “[1988] method of comparing areas under correlated ROC curves as implemented in the pROC package [Robin et al., 2011].”