机器学习基石资源-01_Motivation_of_Dual_SVM_15-54

SVM deeper
• ‘claim’ some results if details unnecessary
—like how we ‘claimed’ Hoeffding
‘Equivalent’ SVM: based on some dual problem of Original SVM
Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 3/23
National Taiwan University (國立台灣大學資訊工程系)
Hsuan-Tien Lin (NTU CSIE)
Machine Learning Techniques
0/23
Dual Support Vector Machine
Roadmap
1
Embedding Numerous Features: Kernel Models
• demanded: not many (large-margin), but sophisticated
boundary (feature transform) ˜ + 1 variables and N constraints • QP with d ˜ large, or infinite?! :-) —challenging if d
• any ‘violating’ (b , w): max • any ‘feasible’ (b , w): max
+ +
n n
αn (some positive) → ∞
all αn ≥0
αn (all non-positive) =
constraints now hidden in max
Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 5/23
1/23
Dual Support Vector Machine
Motivation of Dual SVM
Non-Linear Support Vector Machine Revisited
Non-Linear Hard-Margin SVM
min
b ,w 1 T 2w w 1
Q=
s. t.
yn (wT zn + b) ≥ 1,
Φ(xn )
aT n = yn
2 3
0 0d ˜
0T ˜ d ; p = 0d ˜ +1 ; Id ˜ 1 zT n ; cn = 1
for n = 1, 2, . . . , N
b w
← QP(Q, p, A, c)
˜
return b ∈ R & w ∈ Rd with gSVM (x) = sign(wT Φ(x) + b)
max L(b, w, α)
≥
all αn ≥0
max
min L(b, w, α)
b ,w Lagrange dual
equiv. to original (primal) SVM
• ‘≥’: weak duality
• ‘=’: strong duality, true for QP if
• convex primal • feasible primal (true if Φ-separable) • linear constraints
≥ min L(b, w, α )
b ,w
because max ≥ any for best α ≥ 0 on RHS, min
b ,w all αn ≥0
max L(b, w, α)
≥
all αn ≥0 b,w
max min L(b, w, α )
Lagrange dual problem because best is one of any Lagrange dual problem: ‘outer’ maximization of α on lower bound of original problem
⇔
Regularization by Minimizing Eaug
min Eaug (w) = Ein (w) +
w
λ T w w N
• C equivalent to some λ ≥ 0 by checking optimality condition
∇Ein (w) + solve ‘easily’
2 3
Combining Predictive Features: Aggregation Models Distilling Implicit Features: Extraction Models
Hsuan-Tien Lin (NTU CSIE)
Machine Learning Techniques
Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 7/23
Dual Support Vector Machine
Lagrange Dual SVM
Strong Duality of Quadratic Programming
min
b ,w
all αn ≥0
Lecture 1: Linear Support Vector Machine
linear SVM: more robust and solvable with quadratic programming
Lecture 2: Dual Support Vector Machine
Motivation of Dual SVM Lagrange Dual SVM Solving Dual SVM Messages behind Dual SVM
Machine Learning Techniques (機器學習技法)
Lecture 2: Dual Support Vector Machine Hsuan-Tien Lin (林軒田) htlin@.tw
Department of Computer Science & Information Engineering
1 2 3 4 1 T 2w w 1 T 2w w 1 T 2w w 1 T 2w w
+ α1 (1 + wT z + b) + α2 (1 + wT z + b) + α1 (1 − wT z − b) + α2 (1 − wT z + b) + α1 (1 + wT z + b) + α2 (1 + wT z − b) + α1 (1 − wT z − b) + α2 (1 − wT z − b)
Hsuan-Tien Lin (NTU CSIE)
Dual Support Vector Machine
Lagrange Dual SVM
Lagrange Dual Problem
for any fixed α with all αn ≥ 0, min
b ,w all αn ≥0
max L(b, w, α)
Dual Support Vector Machine
Hale Waihona Puke Motivation of Dual SVM
Key Tool: Lagrange Multipliers
Regularization by Constrained-Minimizing Ein
min Ein (w) s.t. wT w ≤ C
w
1 2 3 4 1 T 2w w 1 T 2w w 1 T 2w w 1 T 2w w
+ α1 (1 + wT z + b) + α2 (1 + wT z + b) + α1 (1 − wT z − b) + α2 (1 − wT z + b) + α1 (1 + wT z + b) + α2 (1 + wT z − b) + α1 (1 − wT z − b) + α2 (1 − wT z − b)
s.t.
for n = 1, 2, . . . , N
yn (wT zn + b) ≥ 1,
αn (1 − yn (wT zn + b))
constraint
Claim
SVM ≡ min
b ,w all αn ≥0
max L(b, w, α)
all αn ≥0
T = min ∞ if violate ; 1 2 w w if feasible b ,w
how many λ’s as variables? N —one per constraint
Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 4/23
Dual Support Vector Machine
Motivation of Dual SVM
Dual Support Vector Machine
Motivation of Dual SVM
Fun Time
Consider two transformed examples (z1 , +1) and (z2 , −1) with z1 = z and z2 = −z. What is the Lagrange function L(b, w, α) of hard-margin SVM?
—called constraint qualification exists primal-dual optimal solution (b, w, α) for both sides