The Random Subspace Method for Constructing Decision Forests

By Tin Kam Ho. 1998

Much of previous attention on decision trees focuses on the splitting criteria and optimization of tree sizes. The dilemma between overfitting and achieving maximum accuracy is seldom resolved. A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity. The classifier consists of multiple trees constructed systematically by pseudorandomly selecting subsets of components of the feature vector, that is, trees constructed in randomly chosen subspaces. The subspace method is compared to single-tree classifiers and other forest construction methods by experiments on publicly available datasets, where the method's superiority is demonstrated. We also discuss independence between trees in a forest and relate that to the combined classification accuracy.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Analysis of a Random Forests Model (2010, methodology): “Breiman’s ideas were decisively inﬂuenced by the early work of Amit and Ge-man [3] on geometric feature selection, the random subspace method of Ho [27] and the random split selection approach of Dietterich [21].”
Fast Edge Detection Using Structured Forests (2014, methodology): “Throughout our presentation we adopt the notation and terminology of the extensive recent survey by Criminisi et al. [11], somewhat simplified for ease of presentation.”
PolyNet: A Pursuit of Structural Diversity in Very Deep Networks (2016, neutral): “First, ensembles usually perform better than any individual networks therein – this has been repeatedly shown in previous studies [3, 19, 4, 10].”
Random Forests (2001, background): “Ho (1998) has written a number of papers on “the random subspace” method which does a random selection of a subset of features to use to grow each tree.”
Gaussian Ranking by Matrix Factorization (2015, methodology): “It combines the random subspace method [4] with parameter sharing across the models in the (implicit) ensemble, which results in a reduced generalization error.”
Yes, Machine Learning Can Be More Secure! A Case Study on Android Malware Detection (2017, methodology): “To this end, multiple linear classifiers are learned by sampling uniformly from the training set (a technique known as bagging [14]) and by randomly subsampling the feature set, as suggested by the random subspace method [22].”
Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor (2020, background): “set is also the famous random subspace method [47], which has been shown to be simple yet effective.”
Generalized random forests (2016, methodology): “However, unlike us, Meinshausen (2006) does not propose a splitting rule that is tailored to the quantile regression context, and instead builds his forests using plain CART regression splits. Thus, a comparison of our method with that of Meinshausen (2006) provides a perfect opportunity for evaluat”
Statistical Arbitrage in Cryptocurrency Markets (2019, background): “Random forests Breiman (1996, 2001); Ho (1995, 1998) are ensemble learners consisting of many decorrelated decision trees which can be understood as their building blocks.”
A Survey on Multi-view Learning (2013, methodology): “Hotelling (1936) introduced canonical correlation analysis (CCA) to describe the linear relation between two views which aims to compute a low-dimensional”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, background): “The authors of [59] trained a random forest [140] to classify active vs passive object trajectories and predict whether or not an object will be involved in a future action.”
Structure-Based Virtual Screening: From Classical to Artificial Intelligence (2020, background): “…(ANNs) (Ashtawy and Mahapatra, 2018); • Support vector machines (Sengupta and Bandyopadhyay, 2014); • Bayesian techniques (Abdo et al., 2010); • Decision tree (Ho, 1998); • k-nearest neighbors (kNN) (Peterson et al., 2009); • Kohonen’s SOMs and counterpropagation ANNs (Schneider et al., 2009); •…”
Random Forests and Decision Trees (2012, methodology): “Those methods such as Bagging or Random subspaces [5,6] which are made from ensemble of various classifiers and those which use randomization for producing diversity have proven to be very efficient.”
A Survey on ensemble learning under the era of deep learning (2021, methodology): “…points Boosting (Freund and Schapire 1997) Wagging (Bauer and Kohavi 1999) Multiboosting (Webb 2000) Manipulating at feature level Random Subspace (Ho 1998) Manipulating at both data and feature levels Random Forests (Breiman 2001) Rotation Forests (Rodríguez et al. 2006) Others Negative…”