Optimizing F-measure: A Tale of Two Approaches
By Nan Ye, Kian Ming Adam Chai, Wee Sun Lee, Hai Leong Chieu. 2012
135 citations recorded by the source corpus.
F-measures are popular performance metrics, particularly for tasks with imbalanced data sets. Algorithms for learning to maximize F-measures follow two approaches: the empirical utility maximization (EUM) approach learns a classifier having optimal performance on training data, while the decision-theoretic approach learns a probabilistic model and then predicts labels with maximum expected F-measure. In this paper, we investigate the theoretical justifications and connections for these two approaches, and we study the conditions under which one approach is preferable to the other using synthetic and real datasets. Given accurate models, our results suggest that the two approaches are asymptotically equivalent given large training and test sets. Nevertheless, empirically, the EUM approach appears to be more robust against model misspecification, and given a good model, the decision-theoretic approach appears to be better for handling rare classes and a common domain adaptation scenario.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Zachary Chase Lipton
- Peter A. Flach
- Meelis Kull
- Harish Guruprasad Ramaswamy
- Balakrishnan Narayanaswamy
- Ryan M. Rifkin
- Alan Mackey
- Ariel Gordon
Citations in the corpus
- Precision-Recall-Gain Curves: PR Analysis Done Right (2015, background): “, they are unbiased in the limit [9, 11]; (iii) given a model, operating points that are optimal for Fβ can be achieved by thresholding the model’s scores [18]; (iv) a classifier yielding perfectly calibrated posterior probabilities has the property that the optimal threshold for F1 is half the opti”
- Consistent Multiclass Algorithms for Complex Performance Measures (2015, methodology): “More recently, there has been increasing interest in designing consistent algorithms for complex performance measures; however, most of this work has focused on the binary case (Ye et al., 2012; Menon et al., 2013; Koyejo et al., 2014; Narasimhan et al., 2014).”
- Optimal Thresholding of Classifiers to Maximize F1 Measure (2014, background): “More recent work in this direction can be found in [19].”
- Scalable Learning of Non-Decomposable Objectives (2016, background): “a restricted class of models [14] or targeting only specific objectives [5, 10, 16, 18, 20].”