PREDICTIVE LEARNING VIA RULE ENSEMBLES
By Jerome H. Friedman, Bogdan E. Popescu. 2008
1,280 citations recorded by the source corpus.
General regression and classification models are constructed as linear combinations of simple rules derived from the data. Each rule consists of a conjunction of a small number of simple statements concerning the values of individual input variables. These rule ensembles are shown to produce predictive accuracy comparable to the best methods. However, their principal advantage lies in interpretation. Because of its simple form, each rule is easy to understand, as is its influence on individual predictions, selected subsets of predictions, or globally over the entire space of joint input variable values. Similarly, the degree of relevance of the respective input variables can be assessed globally, locally in different regions of the input space, or at individual prediction points. Techniques are presented for automatically identifying those variables that are involved in interactions with other variables, the strength and degree of those interactions, as well as the identities of the other variables with which they interact. Graphical representations are used to visualize both main and interaction effects.
Subfields: Artificial Intelligence, Statistics and Probability, Optimization & Training, Algorithms & Complexity
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Statistics and Probability
- Bin Yu
- Tong Zhang
- Peter Buhlmann
- Cynthia Rudin
- Cun-Hui Zhang
- Gérard Biau
- Nicolai Meinshausen
- Erwan Scornet
In Optimization & Training
- Peter Richtárik
- Tong Zhang
- Jakub Konecný
- Gérard Biau
- Zeyuan Allen-Zhu
- Francis R. Bach
- Nicolai Meinshausen
- Erwan Scornet
In Algorithms & Complexity
- Cynthia Rudin
- Kush R. Varshney
- Dmitry M. Malioutov
- Kuldeep S. Meel
- Ş. Ertekin
- Birol Yüceoglu
- Joung Taek Yoon
- Verónica Bolón-Canedo
Citations in the corpus
- FairSight: Visual Analytics for Fairness in Decision Making (2019, background): “For example, feature-level auditing methods seek to analyze the feature importance by permutation [11] or quantifying feature interaction [6, 13, 25].”
- Iterative random forests to discover predictive and stable high-order interactions (2017, methodology): “Popular statistical and machine-learning methods for detecting interactions among features include decision trees and their ensembles: CART (12), random forests (RFs) (13), Node Harvest (14), Forest Garrote (15), and Rulefit3 (16), as well as methods more specific to gene–gene interactions with cate”
- Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model (2015, background): “…associative classifiers are formed by averaging several rules together, or having the rules each vote on the label and then combining the votes, but the resulting classifier is not generally interpretable [Li, Han and Pei (2001), Yin and Han (2003), Friedman and Popescu (2008), Meinshausen (2010)].”
- Interpreting tree ensembles with inTrees (2014, background): “ables as items. 8. Simplified tree ensemble learner (STEL) The rules extracted from a tree ensemble can be summarized into a rulebased learner, referred to as a simplified tree ensemble learner (STEL). [8] used a linear model summarizing the rules from a tree ensemble. Also, ideas in summarize associa”
- Learning customized and optimized lists of rules with mathematical programming (2018, background): “[38], Yin and Han [74], Friedman and Popescu [23], Meinshausen [45].”
- MLIC: A MaxSAT-Based Framework for Learning Interpretable Classification Rules (2018, methodology): “[22] propose a hierarchical kernel learning approach and [21] use optimization to combine basic Boolean clauses obtained from decision trees.”
- Boolean Decision Rules via Column Generation (2018, neutral): “Rule set models have been generalized to rule ensembles [17, 29, 20], using boosting and linear combination rather than logical disjunction; the interpretability of such models is again not comparable to rule sets.”
- Scalable Bayesian Rule Lists (2016, neutral): “This contrasts with, for instance, Friedman and Popescu (2008), who linearly combine pre-mined rules.”
- Node harvest: simple and interpretable regression and classication (2009, methodology): “In a similar spirit, the ‘Rule Ensemble’ al-gorithm by Friedman and Popescu (2008) simplifies interpretability of tree ensembles by selecting just a few nodes across all trees.”
- iml: An R package for Interpretable Machine Learning (2018, background): “• Interaction effects (Friedman, Popescu, and others 2008): Measuring how strongly features interact with each other in the black box model. iml was designed to provide a class-based and user-friendly way to make black box machine learning models interpretable.”
- Machine Learning Interpretability: A Survey on Methods and Metrics (2019, background): “Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate ”
- Data Science of the People , for the People , by the People : A Viewpoint on an Emerging Dichotomy (2015, background): “However, recent work is revisiting the problem of interpretable learning through principled optimization formulations and achieving predictive performance approaching that of uninterpretable methods [40, 24, 18, 10, 34, 30, 55, 33, 50, 20].”
- An Interpretable Stroke Prediction Model using Rules and Bayesian Analysis (2013, neutral): “Call the rule˜r i .”
- Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2022, background): “Feature importance [17, 19] refers to a class of techniques that are used to assign scores to input features.”
- SIRUS: making random forests interpretable (2019, methodology): “All algorithms were run with their default settings (CART trees are pruned, RuleFit is limited to rule predictors).”
- Benchmarking and survey of explanation methods for black box models (2021, background): “RuleFit [48] is a transparent rule learner that exploits an ensemble of trees.”