Simple Rules for Complex Decisions

By Jongbin Jung, Connor Concannon, Ravi Shroff, Sharad Goel, Daniel G. Goldstein. 2017

107 citations recorded by the source corpus.

From doctors diagnosing patients to judges setting bail, experts often base their decisions on experience and intuition rather than on statistical models. While understandable, relying on intuition over models has often been found to result in inferior outcomes. Here we present a new method-select-regress-and-round-for constructing simple rules that perform well for complex decisions. These rules take the form of a weighted checklist, can be applied mentally, and nonetheless rival the performance of modern machine learning algorithms. Our method for creating these rules is itself simple, and can be carried out by practitioners with basic statistics knowledge. We demonstrate this technique with a detailed case study of judicial decisions to release or detain defendants while they await trial. In this application, as in many policy settings, the effects of proposed decision rules cannot be directly observed from historical data: if a rule recommends releasing a defendant that the judge in reality detained, we do not observe what would have happened under the proposed action. We address this key counterfactual estimation problem by drawing on tools from causal inference. We find that simple rules significantly outperform judges and are on par with decisions derived from random forests trained on all available features. Generalizing to 22 varied decision-making domains, we find this basic result replicates. We conclude with an analytical framework that helps explain why these simple decision rules perform as well as they do.

Subfields: Artificial Intelligence, Economics and Econometrics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Optimization & Training

Citations in the corpus

Interpretability via Model Extraction (2017, background): “Recent advances in machine learning have revolutionized our ability to use data to inform critical decisions, such as medical diagnosis [8, 19, 27], bail decisions for defendants [16, 17], and aircra collision avoidance systems [25].”
Algorithmic Decision Making and the Cost of Fairness (2017, background): “Experts increasingly rely on algorithmic decision aids in diverse settings, including law enforcement, education, employment, and medicine [5, 8, 19, 20, 25].”
Manipulating and Measuring Model Interpretability (2018, background): “Examples include generative additive models [3, 18, 19] and point systems [10, 27].”
Interpreting Blackbox Models via Model Extraction (2017, background): “Recent advances in machine learning have revolutionized our ability to use data to inform critical decisions, such as medical diagnosis [30, 16, 44], bail decisions for defendants [28, 27] and the design of aircraft collision avoidance systems [42].”
How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation (2018, methodology): “…2006, Bucilu et al., 2006, Ustun and Rudin, 2016, Doshi-Velez et al., 2015, Kim et al., 2015b, Krakovna and Doshi-Velez, 2016, Hughes et al., 2016, Jung et al., 2017], various kinds of logic-based methods [Wang and Rudin, 2015, Lakkaraju et al., 2016, Singh et al., 2016, Liu and Tsang, 2016,…”
Optimal randomized classification trees (2021, background): “Extracting knowledge from data is a crucial task in Statistics and Machine Learning, which has applications in areas such as Biomedicine [1, 2], Criminal Justice [3, 4, 5], Fraud Detection [6], Privacy Protection [7], Health Care [8, 9, 10, 11], Risk Management [12, 13], Social Networks [14], and Co”
Lessons for artificial intelligence from the study of natural stupidity (2019, methodology): “Work on bail decisions has used sophisticated statistical techniques to infer the probable outcome of release for defendants who were held before trial 38,52 .”
Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning (2020, background): “, simple point systems [24, 59], GAMs [8]) and post-hoc explanation methods that help make blackbox models interpretable (e.”
A large-scale analysis of racial disparities in police stops across the United States (2020, background): “More ambitiously, departments could use their data to design statistically informed guidelines that encourage more consistent, efficient, and equitable decisions [8, 11, 12, 15].”
Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making (2021, background): “Model-specic methods often involve the development of intrinsically interpretable models such as generalized additive models and decision sets [15, 34, 43, 66], as well as visualizing what deep neural network has learned in its interme-”
The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning (2018, background): “In practice, there is often diminishing returns to information, with a relatively short list of key features providing most of the predictive power (Jung et al., 2017), at least partially mitigating this concern.”
An Evaluation of the Human-Interpretability of Explanation (2019, methodology): “ [Caruana et al., 2015, Kim et al., 2015a, Ruping, 2006, Bucilu et al., 2006, Ustun and Rudin, 2016, Doshi-Velez et al., 2015, Kim et al.,¨ 2015b, Krakovna and Doshi-Velez, 2016, Hughes et al., 2016, Jung et al., 2017], various kinds of logic-based methods [Wang and Rudin, 2015, Lakkaraju et al., 20”
Sparsity in Optimal Randomized Classification Trees (2020, background): “Since they are rule-based, when small-sized, they are deemed to be leaders in terms of interpretability [1, 2, 9, 15, 17, 23, 27, 28, 32, 36].”
Economics, Fairness and Algorithmic Bias (2020, unknown): “Examples include point systems (Jung et al., 2017; Ustun and Rudin, 2016), additive models (Caruana et al., 2015; Lou et al., 2012, 2013) or checklists (Malioutov et al., 2017).”
Algorithmic Risk Assessment in the Hands of Humans (2019, unknown): “In the criminal justice context, Jung et al. (2017) show that “simple rules that consider only two features – age and prior FTAs [failureto-appear in court] – perform nearly identically to state-of-the-art machine learning models (random forest and lasso regression) that incorporate all 64…”
Predictive Analytics for City Agencies: Lessons from Children's Services (2017, unknown): “22 My next recommendation would, therefore, be to use simple, interpretable models when possible—subject to performance requirements— rather than complicated black-box models, for their transparency and ease of communication.”