The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation

By Davide Chicco, Niklas Tötsch, Giuseppe Jurman. BioData Mining, 2021

793 citations recorded by the source corpus.

Evaluating binary classifications is a pivotal task in statistics and machine learning, because it can influence decisions in multiple areas, including for example prognosis or therapies of patients in critical conditions. The scientific community has not agreed on a general-purpose statistical indicator for evaluating two-class confusion matrices (having true positives, true negatives, false positives, and false negatives) yet, even if advantages of the Matthews correlation coefficient (MCC) over accuracy and F1 score have already been shown.In this manuscript, we reaffirm that MCC is a robust metric that summarizes the classifier performance in a single value, if positive and negative cases are of equal importance. We compare MCC to other metrics which value positive and negative cases equally: balanced accuracy (BA), bookmaker informedness (BM), and markedness (MK). We explain the mathematical relationships between MCC and these indicators, then show some use cases and a bioinformatics scenario where these metrics disagree and where MCC generates a more informative response.Additionally, we describe three exceptions where BM can be more appropriate: analyzing classifications where dataset prevalence is unrepresentative, comparing classifiers on different datasets, and assessing the random guessing level of a classifier. Except in these cases, we believe that MCC is the most informative among the single metrics discussed, and suggest it as standard measure for scientists of all fields. A Matthews correlation coefficient close to +1, in fact, means having high values for all the other confusion matrix metrics. The same cannot be said for balanced accuracy, markedness, bookmaker informedness, accuracy and F1 score.

Subfields: Artificial Intelligence, Information Systems, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/1c1fd038ee2388faae6bbf4e751b45a28242b982

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Davide Chicco
2. Giuseppe Jurman
3. Matthijs J. Warrens
4. Oguzhan Gencoglu

In Information Systems

1. Giuseppe Jurman
2. Davide Chicco
3. Matthijs J. Warrens
4. Oguzhan Gencoglu

In Molecular Biology

1. Giuseppe Jurman
2. Davide Chicco
3. Matthijs J. Warrens
4. Oguzhan Gencoglu

Citations in the corpus

• The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification (2023, methodology): “In a previous study [4], we defined basic rates for confusion matrices as the following four rates: sensitivity (Eq.”
• Metrics Reloaded: Recommendations for image analysis validation (2022, neutral): “[23] Davide Chicco, Niklas Tötsch, and Giuseppe Jurman.”
• The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment (2021, background): “The Matthews correlation coefficient [1], in particular, is a rate that resulted being more informative than confusion entropy (CEN) [2], accuracy and F1 score [3], balanced accuracy, bookmaker informedness, and markedness [4], and diagnostic odds ratio [5] in the past (Supplementary Information).”