Robust Classification for Imprecise Environments

By Foster J. Provost, Tom Fawcett. 2000

1,377 citations recorded by the source corpus.

In real-world environments it usually is difficult to specify target operating conditions precisely, for example, target misclassification costs. This uncertainty makes building robust classification systems problematic. We show that it is possible to build a hybrid classifier that will perform at least as well as the best available classifier for any target conditions. In some cases, the performance of the hybrid actually can surpass that of the best known classifier. This robust performance extends across a wide variety of comparison frameworks, including the optimization of metrics such as accuracy, expected cost, lift, precision, recall, and workforce utilization. The hybrid also is efficient to build, to store, and to update. The hybrid is based on a method for the comparison of classifier performance that is robust to imprecise class distributions and misclassification costs. The ROC convex hull (ROCCH) method combines techniques from ROC analysis, decision analysis and computational geometry, and adapts them to the particulars of analyzing learned classifiers. The method is efficient and incremental, minimizes the management of classifier performance data, and allows for clear visual comparisons and sensitivity analyses. Finally, we point to empirical evidence that a robust hybrid classifier indeed is needed for many real-world problems.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Statistical Learning Theory

Citations in the corpus

Conscientious Classification: A Data Scientist's Guide to Discrimination-Aware Classification (2017, neutral): “From there it is a matter of execution, and we have offered a process to guide the data scientist, as well as a brief survey of methods as a starting point to consider what is right for the ﬁrm and the application.”
Box drawings for learning with imbalanced data (2014, methodology): “Evaluation Metric We chose to use the area under the convex hull of the ROC curve (AUH) (Provost and Fawcett 2001) as our evaluation metric; it is frequently used for imbalanced classiﬁcation problems and considers the full ROC curve (Receiver Operator Characteristic) curve to evaluate performance.”
FewRel 2.0: Towards More Challenging Few-Shot Relation Classification (2019, background): “, 2006) and minimax estimators (Provost and Fawcett, 2001).”
Learning When Training Data are Costly: The Effect of Class Distribution on Tree Induction (2003, methodology): “In this section we describe the data sets analyzed in this article, the sampling strategy used to alter the class distribution of the training data, the classifier induction program used, and, finally, the metrics for evaluating the performance of the induced classifiers.”
Adversarial classification (2004, background): “(Feature selection can also be carried out online, but we do not pursue that approach here.)”
An Empirical Study of Learning from Imbalanced Data Using Random Forest (2007, methodology): “An in-depth explanation of ROC curves and their potential use for creating optimal classifiers is provided in Provost and Fawcett [16].”
Addressing bias in machine learning algorithms: A pilot study on emotion recognition for intelligent systems (2017, background): “There have been a number of efforts that have tried to resolve this issue with training on imbalanced datasets [1], including using different forms of re-sampling [2], adjusting the decision threshold [3], or a mixture-of-experts approach which combines the results of many classifiers [4].”
ROC analysis: applications to the classification of biological sequences and 3D structures (2008, background): “Subsequently, it gained popularity in machine learning and data mining [7, 8].”
Editorial: special issue on learning from imbalanced data sets (2004, background): “Some of the major papers on this topic that came out at the time of or since the rst workshop are: [ 44 ][18][13][20].”
A Survey of Link Prediction in Social Networks (2011, methodology): “These methods include the altering of the training sample by upsampling or down-sampling [12], altering the learning method by making the process active [16] or cost-sensitive [28], and also more generally by treating the classio er score with different thresholds [ 48 ].”
The Geometry of ROC Space: Understanding Machine Learning Metrics through ROC Isometrics (2003, background): “Provost and Fawcett (2001) use isometrics (which they call isoperformance lines) to determine the optimal point on a ROC convex hull.”
RUSBoost: Improving classification performance when training data is skewed (2008, methodology): “Therefore, we employ Receiver Operating Characteristic curves [7], or ROC curves, to evaluate the models constructed in this study.”
Feature bagging for outlier detection (2005, methodology): “Outlier detection algorithms are typically evaluated using the detection rate, the false alarm rate, and the ROC curves [26].”
On Ensuring that Intelligent Machines Are Well-Behaved (2017, methodology): “Robust optimization algorithms have been proposed as a means for ensuring that machine learning algorithms, ranging from supervised learning algorithms [Provost and Fawcett, 2001] to reinforcement learning algorithms [Garcıa and Fernández, 2015], are in some way “safe” to use.”
Learning Bayesian network classifiers by maximizing conditional likelihood (2004, background): “For example, we may wish to rank cases by probability of class membership (Provost & Domingos, 2003), or the costs associated with incorrect predictions may be variable and not known precisely at learning time (Provost & Fawcett, 2001).”