Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)

By Leo Breiman. 2001

3,901 citations recorded by the source corpus.

There are two cultures in the use of statistical modeling to reach conclusions from data. One assumes that the data are generated bya given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown. The statistical communityhas been committed to the almost exclusive use of data models. This commit- ment has led to irrelevant theory, questionable conclusions, and has kept statisticians from working on a large range of interesting current prob- lems. Algorithmic modeling, both in theoryand practice, has developed rapidlyin fields outside statistics. It can be used both on large complex data sets and as a more accurate and informative alternative to data modeling on smaller data sets. If our goal as a field is to use data to solve problems, then we need to move awayfrom exclusive dependence on data models and adopt a more diverse set of tools.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “Previously, Breiman (2001) referred to similar sets of equivalently performant models as “Rashomon sets”.”
Born Again Neural Networks (2018, background): “In a well-known paper on algorithmic modeling [1], Leo Breiman noted that different stochastic algorithmic procedures [2, 3, 4] can lead to diverse models with similar validation performances.”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “interpretable ML; instead of approximating a known function (a black box ML model), interpretable ML can choose from a potential myriad of approximatelyequally-good models, which, as we noted earlier, is called “the Rashomon set” [41, 97, 269].”
Machine Learning Methods That Economists Should Know About (2019, methodology): “In the abstract of his provocative 2001 paper in Statistical Science the Berkeley statistician Leo Breiman writes about the difference between model-based versus algorithmic approaches to statistics: “There are two cultures in the use of statistical modeling to reach conclusions from data. One assum”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, background): “They present the mathematical problems in interpretable ML so-called ‘‘the multiplicity of good models’’ [60].”
Interpreting Models via Single Tree Approximation (2016, methodology): “Random forests [Breiman, 2001] and other machine learning methods have been widely used across different disciplines and are acknowledged for their outstanding predictive power [Caruana and Niculescu-Mizil, 2006].”
Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model (2015, background): “0 [Quinlan (1993)], CART [Breiman et al. (1984)], ℓ1-regularized logistic regression, support vector machines [Vapnik (1995)], random forests [Breiman (2001a)], and Bayesian CART [Denison, Mallick and Smith (1998), Chipman, George and McCulloch (1998)].”
Peeking Inside the Black Box: Visualizing Statistical Learning With Plots of Individual Conditional Expectation (2013, methodology): “Breiman (2001b), summarizing, stated “accuracy generally requires more complex prediction methods . . .[and] simple and interpretable functions do not make the most accurate predictors.””
Interpretable classification models for recidivism prediction (2015, methodology): “There has been some controversy as to whether sophisticated machine learning methods (such as random forests, Breiman 2001b, Berk et al. 2009, Ritter 2013) are necessary to produce accurate predictive models of recidivism, or if traditional approaches such as logistic regression would sufﬁce (see,…”
Forward stability and model path selection (2021, neutral): “The fact that in any given data modeling context there may exist many near-optimal models was a point stressed by Leo Breiman in his 2001 “Two Cultures” essay (Breiman 2001) where he refers to the phenomenon as the “Rashomon Effect.””
All Models are Wrong, but Many are Useful: Learning a Variable’s Importance by Studying an Entire Class of Prediction Models Simultaneously (2018, methodology): “Our estimator for model reliance differs from the traditional RF VI measure (Breiman, 2001) in that we permute inputs to the overall model, rather than permuting the inputs to each individual ensemble member.”
Estimation and Inference of Heterogeneous Treatment Effects using Random Forests (2015, methodology): “…statistical ideas, but did not establish any formal justiﬁcation for it. b Instead of using subsampling, Breiman originally described random forests in terms of bootstrap sampling, or bagging [Breiman, 1996], where the Z ∗ bi iid ∼ { Z 1 , ..., Z n } form a bootstrap sample of the training data.”
Deep Learning in Finance (2016, background): “Z((1)) = f (1) ( W X + b ) , Z((2)) = f (2) ( W ((1))Z((1)) + b((1)) ) , .”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “3The existence of multiple equally costly recourse actions is commonly referred to as the Rashoman effect [33].”
A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results (2018, methodology): “While the computation and interpretation of their “empirical model class reliance” measure of variable importance could be viewed as similar to those of hacking intervals, their ultimate goal is to study the population version of this quantity, in order to study the Rashomon set for the population.”