Optimal Thresholding of Classifiers to Maximize F1 Measure

By Zachary Chase Lipton, Charles Peter Elkan, Balakrishnan Narayanaswamy. 2014

449 citations recorded by the source corpus.

This paper provides new insight into maximizing F1 measures in the context of binary classification and also in the context of multilabel classification. The harmonic mean of precision and recall, the F1 measure is widely used to evaluate the success of a binary classifier when one class is rare. Micro average, macro average, and per instance average F1 measures are used in multilabel classification. For any classifier that produces a real-valued output, we derive the relationship between the best achievable F1 value and the decision-making threshold that achieves this optimum. As a special case, if the classifier outputs are well-calibrated conditional probabilities, then the optimal threshold is half the optimal F1 value. As another special case, if the classifier is completely uninformative, then the optimal behavior is to classify all examples as positive. When the actual prevalence of positive examples is low, this behavior can be undesirable. As a case study, we discuss the results, which can be surprising, of maximizing F1 when predicting 26,853 labels for Medline documents.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Does mitigating ML's disparate impact require disparate treatment? (2017, background): “To optimize F1 score, the optimal threshold depends on the classifier’s confidence [LEN14].”
Learning to Diagnose with LSTM Recurrent Neural Networks (2015, methodology): “Our proposed LSTM RNN uses memory cells with forget gates (Gers et al., 2000) but without peephole connections (Gers et al., 2003).”
Modeling Missing Data in Clinical Time Series with RNNs (2016, methodology): “We report both micro-averaged (calculated across all predictions ) and macro-averaged (calculated separately on each label, then averaged) measures to mitigate the weaknesses in each (Lipton et al., 2014).”
The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation (2020, unknown): “In multiclass cases, researchers can employ the F 1 micro/macro averaging procedure [55–60], which can be even targeted for ad-hoc optimization [61].”
Artificial Intelligence for Mental Health and Mental Illnesses: An Overview (2019, methodology): “An F1 score is a measure of an algorithm ’ s accuracy that conveys the balance between precision and recall, calculated as 2 * ((precision * recall)/(precision + recall)) [92].”
A Critical Review of Recurrent Neural Networks for Sequence Learning (2015, methodology): “Even in the straightforward case of binary classification, without time dependencies, commonly used performance metrics like F1 give rise to complicated thresholding strategies which may not accord with any reasonable intuition of what should constitute good performance [Lipton et al., 2014].”
Directly Modeling Missing Data in Sequences with RNNs: Improved Classification of Clinical Time Series (2016, methodology): “For AUC and F1, we report both micro-averaged (calculated across all predictions) and macro-averaged (calculated separately across on each labels) measures to mitigate the weaknesses in each (Lipton et al., 2014).”
Precision-Recall-Gain Curves: PR Analysis Done Right (2015, background): “…F β can be achieved by thresholding the model’s scores [18]; ( iv ) a classiﬁer yielding perfectly calibrated posterior probabilities has the property that the optimal threshold for F 1 is half the optimal F 1 at that point (ﬁrst proved by [20] and later by [10], while generalised to F β by [9]).”