Adaboost

Weights Increased
Boosting illustration
Weak Classifier 2
Boosting illustration
Weights Increased
Boosting illustration
Weak Classifier 3
Boosting illustration
1 Initialization: D1 (i) m , i 1,, m
Dt (i):probability distribution of xi 's at time t
For t 1,, T :
• Find classifier ht : X {1, 1} which minimizes error wrt Dt ,i.e.,
Given: ( x1 , y1 ),,( xm , ym ) where xi X , yi {1, 1}
1 Initialization: D1 (i) m , i 1,, m
For t 1,, T :
• Find classifier ht : X {1, 1} which minimizes error wrt Dt ,i.e.,
Minimize exponential loss
lossexp H ( x) Ex , y e
yH ( x )

Goal
T Final classifier: sign H ( x) t ht ( x) t 1
Minimize exponential loss
ht arg min j where j Dt (i)[ yi h j ( xi )]
hj
m
• Weight classifier: t
1 1 t ln 2 t Dt (i) exp[t yi ht ( xi )] , Zt is for normalization Zt
Given: ( x1 , y1 ),,( xm , ym ) where xi X , yi {1, 1}
1 Initialization: D1 (i) m , i 1,, m
For t 1,, T :
• Find classifier ht : X {1, 1} which minimizes error wrt Dt ,i.e.,
T Final classifier: sign H ( x) t ht ( x) t 1
Define Ht ( x) Ht 1 ( x) t ht ( x) with H 0 ( x) 0
Ex , y e yHt ( x ) Ex E y e yHt ( x ) | x Ex E y e y[ Ht 1 ( x )t ht ( x )] | x Ex E y e yHt 1 ( x ) e yt ht ( x ) | x Ex e yHt 1 ( x ) et P( y ht ( x)) et P( y ht ( x))
Final classifier is a combination of weak classifiers
AdaBoost & Its Applications
How and why AdaBoost works?
What goal the AdaBoost wants to reach?
The AdaBoost Algorithm
• Update distribution: Dt 1 (i)
Dt (i) exp[t yi ht ( xi )] , Zt is for normalization Zt Give error classified patterns more chance for learning.
The AdaBoost Algorithm
i 1
• Update distribution: Dt 1 (i)
T Output final classifier: sign H ( x) t ht ( x) t 1
Goal
T Final classifier: sign H ( x) t ht ( x) t 1
lossexp H ( x) Ex , y e
yH ( x ) yH

Maximize the margin yH(x)
Minimize
lossexp H ( x) Ex , y e yH ( x )
Goal
Then, H ( x) HT ( x)
ht arg min j where j Dt (i)[ yi h j ( xi )]
hj
m
• Weight classifier: t
1 1 t ln 2 t Dt (i) exp[t yi ht ( xi )] , Zt is for normalization Zt
T Final classifier: sign H ( x) t ht ( x) t 1
Define Ht ( x) Ht 1 ( x) t ht ( x) with H 0 ( x) 0
1 1 t 1 P( y ht ( x)) t ln t ln 2 t 2 P( y ht ( x))
Given: ( x1 , y1 ),,( xm , ym ) where xi X , yi {1, 1}
1 Initialization: D1 (i) m , i 1,, m
For t 1,, T :
They are goal dependent.
• Find classifier ht : X {1, 1} which minimizes error wrt Dt ,i.e.,
t ?
Then, H ( x) HT ( x)
Minimize
lossexp H ( x) Ex , y e yH ( x )
T Final classifier: sign H ( x) t ht ( x) t 1
Define Ht ( x) Ht 1 ( x) t ht ( x) with H 0 ( x) 0

. . .
HT ( x) sign t ht ( x) How to t 1
– – – – –
Each weak classifier learns by considering one simple feature T most beneficial features for classification should be selected T
. . .
T HT ( x) sign t ht ( x) t 1
hT ( x) {1, 1}
weak classifiers
slightly better than random
strong classifier
Weaker Classifiers
h1 ( x) {1, 1} h2 ( x) {1, 1}
ht arg min j where j Dt (i)[ yi h j ( xi )]
hj
m
minimize weighted error
• Weight classifier: t
1 1 t ln 2 t
i 1
for minimize exponential loss
AdaBoost & Its Applications
主講人：虞台文
Outline

Overview The AdaBoost Algorithm How and why AdaBoost works? AdaBoost for Face Detection
AdaBoost & Its Applications
i 1
• Update distribution: Dt 1 (i)
T Output final classifier: sign H ( x) t ht ( x) t 1
Boosting illustration
Weak Classifier 1
Boosting illustration
Ex e yHt 1 ( x ) et P( y ht ( x)) et P( y ht ( x)) 0
0
t ?
Then, H ( x) HT ( x)
Minimize
lossexp H ( x) Ex , y e yH ( x )
The Strong Classifiers
h1 ( x) {1, 1T ( x) sign t ht ( x) t 1
hT ( x) {1, 1}
weak classifiers
slightly better than random
Overview
Introduction
AdaBoost
Adaptive Boosting
A learning algorithm
Building a strong classifier a lot of weaker ones
AdaBoost Concept
h1 ( x) {1, 1} h2 ( x) {1, 1}
ht arg min j where j Dt (i)[ yi h j ( xi )]
hj
m
• Weight classifier: t