bagging

Bagging1Perturb and combineDecision trees are unstable models.That is,small changes in the training data can cause large changes in prediction.However,the overall performance of the tree remains stable(Breiman et al.,1984).For example,changing the class label of one case may result in a completely different tree with nearly the same accuracy.The instability can be explained as follows.At each split,there are typically a number of splits on the same and different inputs that give similar performance (competitive splits).The competition among them is so tough that a small change in the data can easily result in a different split being chosen.This in turn produces different subsets in the child nodes;the changes in the data are even larger in the child nodes.The changes continue to cascade down the ually local modeling methods(e.g.trees)tend to be more unstable and sensitive to small changes than global methods(e.g.linear models).Methods have been devised to take advantage of the instability of trees to cre-ate models that are more powerful.Perturb and combine(P&C)methods generate multiple models by manipulating the distribution of the data or altering the con-struction method and then averaging the results(Breiman,1998).Some perturbation methods:•Resample•Subsample•Add noise•Adaptively reweight•Randomly choose from among the competitor splitsAn ensemble model is the combination of multiple models.The combinations can be formed by•For classification:voting(weighted or unweighted)on the classifications •For regression:averaging(weighted or unweighted)of the predicted values.66Ensemble methods are a very active area of research in thefield of machine learning and statistics.Bagging,boosting,and random forests all fall into P&C methods.The attractiveness of P&C methods is their improved performance over single models(though the simple interpretation of a single tree is lost).Bauer and Kohavi (1999)demonstrated the superiority of P&C methods with extensive experimenta-tion.One reason why simple P&C methods give improved performance is variance reduction.If the base models have low bias and high variance,then averaging decreases the variance.In contrast,combining stable models can negatively affect performance.2Bootstrapping•Empirical distribution functionˆF(x)=1n ni=1I(X i≤x)where the indicator function I(·)is defined as1if X i≤x and0if X i>x. The empirical distribution function assigns mass1/n for each observed X i.•Bootstrap sampleA sample of size n from the empirical distribution functionˆF(x)is called a bootstrap sample.A bootstrap sample is generated by sampling with replace-ment from(X1,...,X n).•Bootstrap estimatesSuppose we have a sample(X1,...,X n)from F(unknown),what is the stan-dard error of its interquartile range(IQR)?If we can sample repeatedly from F,we can estimate IQR based on each sample.And we can in turn calculate the SE using the repeated realizations of IQR.In reality,we can not take repeated samples from F.However,we have an estimateˆF of F and we can,hence,sample repeatedly fromˆF.So here is how a bootstrap estimate of the SE of IRQ may be calculated:67–For i=1to B(number of bootstrap samples){–Take a bootstrap sample of size n:(X 1,...,X n)–Calculate the IRQ of(X 1,...,X n),IRQ i–}–The bootstrap estimate of the SE of IRQ is given by:SE =sample standard deviation of(IRQ i,i=1,...,B)•When might the bootstrap fail?Bootstrap might fail when the empirical distributionˆF does not represent F well,for example,data with outliers.For survival data,both the observation time and the failure indicator should form the sampling unit.For dependent data,each cluster(not each subject within clusters)should form the sampling unit.3BaggingBagging(bootstrap aggregating)is the original P&C method(Breiman,1996). 1.Draw B bootstrap samples.A bootstrap sample is a random sample of sizen drawn from the empirical distribution of a sample of size n;that is,the training data is resampled with replacement.Some of the cases will be left out of the sample(approximately36.8%in each bootstrap sample)and some cases will be represented more than once.2.Build a tree on each bootstrap sample.Pruning can be counterproductive(Bauer and Kohavi,1999).Large trees with low bias and high variance are ideal.3.Thefinal aggregate classifier can be obtained by averaging(regression)ormajority voting(classification).Breiman(1996)used50bootstrap replications for classification and25for regres-sion.Bauer and Kohavi(1999)used25replications for both voting and averaging.68Variance Reduction•Bagging reduces variance:Each single classifier is unstable-that is,it has high variance.The aggregated classifier has a smaller variance than a single original classifier.•Bagging works well for unstable learning algorithms.Bagging can slightly degrade the performance of stable learning algorithms.–Unstable learning algorithms:small changes in the training set result inlarge changes in predictions.For example,Neural network;Decision treesand Regression trees;Subset selection in linear/logistic regression–Stable learning algorithms:K-nearest neighborsBagging Can Hurt•Bagging helps when a learning algorithm is good on average but unstable with respect to the training set.But if we bag a stable learning algorithm,we can actually make it worse.•Bagging almost always helps with regression,but even with unstable learners it can hurt in classification.If we bag a poor and stable classifier we can make it horrible.“The vital element is the instability of the prediction method.If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy.”(Breiman,1996)Bagging can be implemented in R using the ipred package.4References•Bauer,E.and Kohavi,R.(1999).An Empirical Comparison of Voting Classi-fication Algorithms:Bagging,Boosting,and Variants.Machine Learning,36, 105-142.•Breiman,L.(1996).Bagging predictors.Machine Learning,24(2):123-140.69•Breiman,L.(1998).Arcing classifiers.The Annals of Statistics,26(3),801–849.•Efron,B.and Tibshirani,R.(1993).An Introduction to the Bootstrap.Chap-man and Hall.70。