Fisher线性分类器

2013-9-12
ICL,wenzs@
2
Classification Problem

Classification Problem:

Mapping M: X->C X is the Instances of the problem C is the known set of class
17
Strategy to determine w0

Several advised method
y0=(m1’+ m2’)/2 y0=(N1m1’+ N2m2’)/(N1+N2)

……
2013-9-12
ICL,wenzs@
18
FDF


VS
SVM
Fisher’s Linear Classification
ICL,wenzs@
7
2-Classes


Simple->Complex

Consider 2-Classes problem
2-Classes problem
X->{0,1} Concept learning

2013-9-12
ICL,wenzs@
8

=> Sbw* –λSωw*=0 => Sbw* =λSωw* => Sω-1Sbw* =λw*

1. The eigenvector of Sω-1Sb 2. w*= Sω-1(m1-m2)

2013-9-12 ICL,wenzs@ 15
The best “w”

Optimization problem

Mathematic solution
Lagrange Lemma of Maximization

2013-9-12
ICL,wenzs@
14
Lagrange method
Let wTSωw=c≠0 L(w,λ)= wTSbw –λwTSωw

Classifier’s task

(narrow sense)
Given a classified sample X={x1,…,xN} and C={c1,…,ck}, Mapping? Given a unclassified sample X and the count of classes
ICL,wenzs@ 11
2013-9-12
Numerical Characters


Class Mean Vector
Inter Divergence Matrix

Class Divergence Matrix
2013-9-12 ICL,wenzs@ 12
2013-9-12 ICL,wenzs@ 9
2013-9-12
ICL,wenzs@
10
Fisher’s Discriminant

Fisher’s Discriminant Function (FDF)

Fisher’s Principle

Maximize JF(w)

Feature Vector
For every x∈X, x is represented as vector (x1,…,xn) xi (1 ≤ i ≤ n) is called feature of x

2013-9-12 ICL,wenzs@ 3
Classifier

Cluster


cluster unconsidered in this report
2013-9-12
ICL,wenzs@
4
Linear Classifier

Find n linear function gi(x) i=1,…,n

M(x)= argmax{gi(x) }
High dimension -> 1-D w*->ok, what about w0?

w0 is a threshold g(x)= w*x+w0

If g()＞0, x∈C1; If g(x)＜0, x∈C2 If g(x)=0?
2013-9-12
ICL,wenzs@
Maximize JF(w)

Linear SVM

Maximize “margin”
2013-9-12
ICL,wenzs@
19
2013-9-12
ICL,wenzs@
20
Conclusion
Provide an approach to transform problem from high dimension to 1-D linear space Limited to linear discriminant Simple but still used
Fisher’s Linear Classifier
Machine Learning Seminar
2013-9-12
ICL,wenzs@
1
Content
Classifier, Linear Classifier Fisher’s Discriminant Function The best project vector Comparison with Linear SVM Conclusion
i
2013-9-12
ICL,wenzs@
5
Project Vector
2013-9-12
ICL,wenzs@
6
Discriminant Function

Different Criterion
Fisher Perceptron SVM ……

2013-9-12
Transformation
y1-y2=(wTx1+w0)-(wTx2+w0) = wT(x1-x2) (m1’-m2’)2= wTSbw D12+D22=wT(S1+S2)w= wTSωw

2013-9-12
ICL,wenzs@
13
Find the best “w”

Lemma of Maximization

/yujs/papers/ pdf/max.pdf w*=Sω-1(m1-m2)

Despite of scale, w* is

2013-9-12
ICL,wenzs@
16
Remain Problem

2013-9-12
ICL,wenzs@
21
Ronald Aylmer Fisher
(1890-1962)




Brilliant biologists Development of methods suitable for small samples Discovery of the precise distributions of many sample statistics Invention of analysis of variance.