随机森林原理及应用

over-fitting

Bagging Boosting N( C4.5

Bagging

Bagging Bootstrap AGGregatING Breiman Bagging

Bagging Bagging 63.2% Breiman Bagging Bagging Bagging Bagging Bagging Bagging Bagging Breiman

Bagging Boosting

Boosting

Boosting Robert T.Schapire …… Boosting

Boosting Valiant PAC(Probably Approxi mately Correct) Valiant Kearns 1/2, ,Valiant Kearns PAC 1990 ,Schapire , Boosting ,Freund Boosting

Boosting Boosting Boosting , Boosting (BP), (C4.5)

Adaboost

Boosting Freund Schapire AdaBoost Freund AdaBoost Boosting ,AdaBoost D(x)t, Ht(x), D(x)t, , 0.15,

,AdaBoost AdaBoost :

AdaBoost

(1) 1/n

(2)

: :

e= (ht(x i) yi)Di(1)

:W t=F(e), AdaBoost boost AdaBoost

1. T

2. 1/n

3.

(1)

(2)

(3)

(4)

Adaboost Boosting

bootsrap bootsrap

2

RF ,RF

1.

2.

3.

4.

5.

6.

7.

8. outlier

9.

10.

11.

12.

13.

1.

2.

3.

4.

(RFC) {(,),1,2,...,}

k h X k N k RFC bootstrap

RF

{(,),1,2,...,}k h X k N :

()arg max (())i Y i

H x I h x Y ,()H x ,()i h x ,Y ),(.)I

-

37% (m <

RF bagging bootstrap bagging 37% bootstrap , (Out Of Bag,OOB) OOB Bagging RFC OOB OOB OOB RF Breiman ,OOB

:

RF ) , CART RF bagging bo ot st rap RF

, ,L [-1,1]

,RF ? Breiman 1000), OOB OOB 4000), Breiman RF

3

, ReliefF CM

1,

REL IEF CM

,

RELIEF

RELIEF

CM(contextual merit)

CM :1 ,r s

,r s ,r s ,r s D ,r s D ,r s ,r s D ,r s

x i x

x

x x

F1

, (TP (TN ), (FN (FP (P)=TP+FN, (N)=FP+TN, (C)=P+N

”

Error rate=1-Accuracy=(FN+FP)/C

(Accuracy) Accuracy= TP+TN /C

(Precision) Precision=TP/(TP+FP)

(Recall)

Recall=TP/P

F1

——ROC AUC

ROC ROC Receiver Operating Characteristic RelativeOperating Characteristic ROC Lusted ROC ROC

(False Positive Rate,FPR)=FP/N

(True Positive Rate.TPR)=TP/P

ROC FPR TPR FPR

TPR

ROC (FPR=0,TPR=0) (FPR=1,TPR=1) (FPR=0,TPR=1)

ROC ROC ROC

ROC ROC

AUC ROC curve AUC 0.5 1.0 AUC

Bagging

Bagging

Bagging Bagging