Classifier Technology and the Illusion of Progress

By David J. Hand. 2006

798 citations recorded by the source corpus.

A great many tools have been developed for supervised clas- sification, ranging from early methods such as linear discriminant anal- ysis through to modern developments such as neural networks and sup- port vector machines. A large number of comparative studies have been conducted in attempts to establish the relative superiority of these methods. This paper argues that these comparisons often fail to take into account important aspects of real problems, so that the apparent superiority of more sophisticated methods may be something of an illu- sion. In particular, simple methods typically yield performance almost as good as more sophisticated methods, to the extent that the di!erence in performance may be swamped by other sources of uncertainty that generally are not considered in the classical supervised classification paradigm.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead (2018, background): “This is related to (but different from) the notation of “flat minima,” for which a nice example is given by Hand [19].”
OpenML: networked science in machine learning (2014, background): “This makes us duplicate a lot of effort, and ultimately slows down the whole field of machine learning [43; 14].”
A Method of Automated Nonparametric Content Analysis for Social Science (2010, background): “As Hand writes, “Of course, it would be a brave person who could confidently assert that these two conditions held” (2006, 9).”
Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research (2015, methodology): “This may indicate that progress in the field has stalled (e.g., Hand, 2006), and that the focus of attention should move from PD On the other hand, we do not expect the desire to develop better, more accurate scorecards to end any time soon.”
Transaction aggregation as a strategy for credit card fraud detection (2009, methodology): “If a method requires a high degree of specialist knowledge and a great deal of time or effort in order to set the parameters, then it is less likely to be successful in practice (Hand 2006).”
Discrimination capacity in species distribution models depends on the representativeness of the environmental domain (2013, background): “Statisticians know that no classification method can be universally advocated, and that the improved performance of new complex techniques may not be as relevant or useful as it may seem at first (Hand, 2006, and references therein).”
Support vector machines for credit scoring and discovery of significant features (2009, methodology): “Hand (2006) points out that for many classification problems, the data suffers rom population drift, in that the class distributions shift over time.”
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts (2013, methodology): “The new organization scheme and documents classified according to those categories is the contribution of the method, not the time difference between applying supervised and unsupervised methods.”
Resolving confusion of tongues in statistics and machine learning: A primer for biologists and bioinformaticians (2012, background): “If the same subjects are used for building the classifier and assessing its performance, the errors will be optimistically small and will not generalize to new data very well [4].”
Fightin' Words: Lexical Feature Selection and Evaluation for Identifying the Content of Political Conflict (2008, background): “Department of Political Science, Quantitative Social Science Initiative, The Pennsylvania State University, e-mail: burtmonroe@psu.edu (corresponding author)”
Machine Learning and Its Applications to Biology (2007, background): “It is argued [39] that the success or failure of machine learning approaches on a given problem is sometimes a matter of the quality indices used to evaluate the results, and these may vary strongly with the expertise of the user.”
Recent developments in consumer credit risk assessment (2007, background): “However (Hand, 2006) argues that the relative performances illustrated in comparisons like Table 2 overemphasise the likely differences in predictive performance when classifiers are used in practice.”