machine learning Machine learning (ML) is a field of study in artificial intelligence concerned with the development and study of Computational statistics, statistical algorithms that can learn from data and generalise to unseen data, and thus perform Task ( ...

the random subspace method, also called attribute bagging or feature bagging, is an

ensemble learning In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statist ...

method that attempts to reduce the

correlation In statistics, correlation or dependence is any statistical relationship, whether causal or not, between two random variables or bivariate data. Although in the broadest sense, "correlation" may indicate any type of association, in statistics ...

between

estimator In statistics, an estimator is a rule for calculating an estimate of a given quantity based on Sample (statistics), observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguish ...

s in an ensemble by training them on random samples of

features Feature may refer to: Computing * Feature recognition, could be a hole, pocket, or notch * Feature (computer vision), could be an edge, corner or blob * Feature (machine learning), in statistics: individual measurable properties of the phenome ...

instead of the entire feature set.

Motivation

In ensemble learning one tries to combine the models produced by several ''learners'' into an ''ensemble'' that performs better than the original learners. One way of combining learners is

bootstrap aggregating Bootstrap aggregating, also called bagging (from bootstrap aggregating) or bootstrapping, is a machine learning (ML) ensemble meta-algorithm designed to improve the stability and accuracy of ML classification and regression algorithms. It also ...

or ''bagging'', which shows each learner a randomly sampled subset of the training points so that the learners will produce different

models A model is an informative representation of an object, person, or system. The term originally denoted the plans of a building in late 16th-century English, and derived via French and Italian ultimately from Latin , . Models can be divided int ...

that can be sensibly averaged. In bagging, one samples training points with replacement from the full training set. The random subspace method is similar to bagging except that the

("attributes", "predictors", "independent variables") are randomly sampled, with replacement, for each learner. Informally, this causes individual learners to not over-focus on features that appear highly predictive/descriptive in the training set, but fail to be as predictive for points outside that set. For this reason, random subspaces are an attractive choice for high-dimensional problems where the number of features is much larger than the number of training points, such as learning from fMRI data or gene expression data. The random subspace method has been used for

decision trees A decision tree is a decision support system, decision support recursive partitioning structure that uses a Tree (graph theory), tree-like Causal model, model of decisions and their possible consequences, including probability, chance event ou ...

; when combined with "ordinary" bagging of decision trees, the resulting models are called

random forest Random forests or random decision forests is an ensemble learning method for statistical classification, classification, regression analysis, regression and other tasks that works by creating a multitude of decision tree learning, decision trees ...

s. It has also been applied to

linear classifier In machine learning, a linear classifier makes a classification decision for each object based on a linear combination of its features. Such classifiers work well for practical problems such as document classification, and more generally for prob ...

support vector machine In machine learning, support vector machines (SVMs, also support vector networks) are supervised max-margin models with associated learning algorithms that analyze data for classification and regression analysis. Developed at AT&T Bell Laborato ...

s, nearest neighbours and other types of classifiers. This method is also applicable to one-class classifiers. The random subspace method has also been applied to the portfolio selection problem showing its superiority to the conventional resampled portfolio essentially based on Bagging. To tackle high-dimensional sparse problems, a framework name
Random Subspace Ensemble (RaSE)
ref name=":0"> was developed. RaSE combines weak learners trained in random subspaces with a two-layer structure and iterative process. RaSE has been shown to enjoy appealing theoretical properties and practical performance.

Algorithm

An ensemble of models employing the random subspace method can be constructed using the following

algorithm In mathematics and computer science, an algorithm () is a finite sequence of Rigour#Mathematics, mathematically rigorous instructions, typically used to solve a class of specific Computational problem, problems or to perform a computation. Algo ...

: # Let the number of training points be ''N'' and the number of features in the training data be ''D''. # Let ''L'' be the number of individual models in the ensemble. # For each individual model ''l'', choose ''n (n < N)'' to be the number of input points for l. It is common to have only one value of n for all the individual models. # For each individual model l, create a training set by choosing ''d ''features from D with replacement and train the model. Now, to apply the ensemble model to an unseen point, combine the outputs of the ''L'' individual models by majority voting or by combining the posterior probabilities.

Footnotes

References

{{DEFAULTSORT:Random Subspace Method Classification algorithms Ensemble learning