The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation
By Davide Chicco, Giuseppe Jurman. BMC Genomics, 2020
5,423 citations recorded by the source corpus.
To evaluate binary classifications and their confusion matrices, scientific researchers can employ several statistical rates, accordingly to the goal of the experiment they are investigating. Despite being a crucial issue in machine learning, no widespread consensus has been reached on a unified elective chosen measure yet. Accuracy and F1 score computed on confusion matrices have been (and still are) among the most popular adopted metrics in binary classification tasks. However, these statistical measures can dangerously show overoptimistic inflated results, especially on imbalanced datasets. The Matthews correlation coefficient (MCC), instead, is a more reliable statistical rate which produces a high score only if the prediction obtained good results in all of the four confusion matrix categories (true positives, false negatives, true negatives, and false positives), proportionally both to the size of positive elements and the size of negative elements in the dataset. In this article, we show how MCC produces a more informative and truthful score in evaluating binary classifications than accuracy and F1 score, by first explaining the mathematical properties, and then the asset of MCC in six synthetic use cases and in a real genomics scenario. We believe that the Matthews correlation coefficient should be preferred to accuracy and F1 score in evaluating binary classification tasks by all scientific communities.
Subfields: Artificial Intelligence, Information Systems
View Paper: https://www.semanticscholar.org/paper/44858b662822d291a5e1e9e3501ad19a9f1572c2
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Oguzhan Gencoglu
- Davide Chicco
- Giuseppe Jurman
- Matthijs J. Warrens
- Wing Yiu Choy
- Margherita Grandini
- Giorgio Visani
- Enrico Bagli
In Information Systems
- Oguzhan Gencoglu
- Giuseppe Jurman
- Davide Chicco
- Matthijs J. Warrens
- Wing Yiu Choy
- Margherita Grandini
- Giorgio Visani
- Enrico Bagli
Citations in the corpus
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (2024, neutral): “Evaluating models based on their real-world use cases is well-supported across various research disciplines [3, 8, 20].”
- Credit Card Fraud Detection using Machine Learning Algorithms (2019, neutral): “5.3 Another used for statistical evaluation of classification performances is Matthews Correlation Coefficient (MCC). determined from the Chicco and Jurman, 2020):”
- TransAttUnet: Multi-Level Attention-Guided U-Net With Transformer for Medical Image Segmentation (2021, methodology): “3) Evaluation Metrics: In our experiments, we adopt the mean Dice coefficient (DICE) [46] and F1-score [47] as the key evaluation metric to measure the extent of similarity between the predicted mask and ground truth.”
- Metrics for Multi-Class Classification: an Overview (2020, methodology): “Only in the 2000s MCC became a widely employed metric to test the performance of Machine Learning techniques with some extensions to the multi-class case [4].”
- Cyberbullying Detection With Fairness Constraints (2020, methodology): “As accuracy, area under the receiver operating characteristic curve, and F1 score may lead to misleading conclusions on imbalanced datasets (such as here), we evaluate our models with Matthews correlation coefficient (MCC) which does not exhibit such drawback [60], [61], [62], [63].”
- An open-source, high-performance tool for automated sleep staging (2021, background): “The latter is thought to be the most robust and informative score for classification as it naturally takes into account imbalance between sleep stages ( Chicco and Jurman, 2020 ).”
- Agricultural Drought Prediction Based on Conditional Distributions of Vine Copulas (2021, neutral): “…both lower than −0.5 in the sample of SSI observations lower than −0.5; and F1S is the harmonic mean value between precision and recall, which is the most popular metric used in binary classification evaluation and helps comprehensively measure the accuracy of a model (Chicco & Jurman, 2020).”
- The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification (2023, methodology): “In some previous studies, we argued that the MCC is more informative than confusion entropy [15], F 1 score [6], accuracy [6], balanced accuracy [4], bookmaker informed-ness [4], markedness [4], diagnostic odds ratio [16], Cohen’s Kappa [17], and Brier score [17].”
- KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization (2021, methodology): “For the performance evaluation metrics Chicco & Jurman (2020), we follow the work of TinyBERTJiao et al. (2019) by setting Matthews Correlation for CoLA, F1 for MRPC and QQP, Spearman Correlation for STS-B, and accuracy for the other tasks.”
- A Machine Learning Approach to Diagnosing Lung and Colon Cancer Using a Deep Learning-Based Classification Framework (2021, background): “Their proposed method does not require the segmentation of nuclei and can provide a maximum F-measure of 80.2% while classifying them.”
- F1 Is Not Enough! Models and Evaluation towards User-Centered Explainable Question Answering (2020, background): “F1 has been criticized from various perspectives including theoretical considerations and concrete applications (Hand and Christen, 2018; Chicco and Jurman, 2020; Sokolova et al., 2006).”
- The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment (2021, methodology): “Finally, κ can be undefined in some cases, but these cases can be handled with mathematical operations similar to the ones needed when MCC is undefined [3].”
- ADMETlab 3.0: an updated comprehensive online ADMET prediction platform enhanced with broader coverage, improved performance, API functionality and decision support (2024, neutral): “…the performance of the models, R-square (R 2 ), root mean squared error (RMSE), and mean absolute error (MAE) were applied for regression tasks, while the area under the ROC curve (AUC), accuracy (ACC), and Matthews correlation coefficient (MCC) ( 39 ) were applied for classification tasks.”
- Applications of Machine Learning in Human Microbiome Studies: A Review on Feature Selection, Biomarker Identification, Disease Prediction and Treatment (2021, methodology): “MCC was used since it is a suitable performance measure to evaluate model performance even in the case of imbalanced classes (Chicco and Jurman, 2020).”
- Boosting methods for multi-class imbalanced data classification: an experimental review (2020, methodology): “To have an intuitive view of the performance of the classifier based on these metrics, F-measure [72] can be used.”
- AlphaFold2: A Role for Disordered Protein/Region Prediction? (2022, methodology): “Thresholds for n were chosen based on the Matthews correlation coefficient (MCC), which has been documented to be an excellent metric for assessing the accuracy of binary classifiers [33] and was the approach used at CAID [29].”