EM Algorithm

2
vz u, v v, v
2
u u
2
v z
2
2
2
u, v v
2
2
z
2
u, v v
2
2
v u, v
n
Here we consider the variant form in Euclidean Space
n 2 n 2 xi yi xi yi i 1 i 1 i 1
• How to construct a convex function?
If f(x) is twice differentiable on [a,b] and f ''( x) 0 on [a,b] then f(x) is convex on [a,b]. ln( x) is strictly convex on (0, ) .Hence optimized object is to maximize log-likelihood function.
2
Mathematical Foundations(cont.)
n 1 holds n 1
n n 1 x y x y x y i i i i n 1 n 1 i 1 i 1 2 2
x y x y
2 2 n 1 n 1
n 2 2 n 2 n 2 n 2 x y x yi yn 1 xi xi yi i 1 i 1 i 1 i 1 n n 2 2 2 xi xn 1 yi2 yn 1 i 1 i 1 n 1 2 n 1 2 xi yi i 1 i 1
Example
• True distribution vs What we see
• Problem: How to estimate the parameter of distribution from the right picture?
Dilemma of Estimation
We can only get the information of distribution if we are sure which sample belongs to which class. We can only get the information of sample belonging if we are sure what shape the distribution is. All are resulted from the existing of hidden variable.
u, v u, v z, v u v, v u, v v, v 0 v, v v, v
Mathematical Foundations(cont.)
z is a vector orthogonal to the vector v
u u, v v, v
x1 y1 xn yn x1 xn
2

2
2
y
2
1
yn
2

Mathematical Foundations(cont.)
• Proof procedure of Cauchy–Schwarz inequality
Let u,v be arbitrary vectors in vector space V over F, where F is the field of real or complex numbers 1. v=0 clearly hold the inequality u , v u v 2. Assume u , v 0 u, v Let z u v v, v
Mathematical Foundations(cont.)
• Let f be a real valued function defined on an interval I=[a,b]. f is said to be convex on I if x1, x2 I , [0,1] f ( x1 (1 ) x2 ) f ( x1 ) (1 ) f ( x2 )
2 2 n 1 n 1
EM mechanism
• Problem Formulation
Given a set of independent observed sample x x1, x2 ,, xn with the dimension of d which observe the P-distribution with i i i i the parameter to be estimated in the vector p p1 , p2 , ps . 1 i k representing there are k similar distribution. Determine the values of i group with the maximized probability by introducing |z| hidden values ( z z1, z2 ,, zk ). ,0,1,0,0, ,0] The meaning of zi can be denoted as [0,0, i 1 k i Let p( zk 1) k
Mathematical Foundations(cont.)
• Proof procedure of Jensen equality
n=1: trivial case; n=2: correspond to the definition of convexity; n≥3: f n1 x f x n x f x (1 )
EM mechanism(cont.)
• The basic structure of EM is as follows:
Initialization: randomly set the component E-step: Determine the conditional expectation EZ X , ln P( X , z ) M-step: Maximize EZ X , ln P( X , z ) w.r.t t 1 stable l ( ) D , Do until convergence: Q , t 1 : E c
n 1 f xn 1 (1 n 1 ) n 1 f xn 1 i f ( xi )
i 1 n
i f ( xi ) 1 i 1 n 1
n
i f xi
i 1
n 1
Mathematical Foundations(cont.)
Mathematical Foundations(cont.)
• Cauchy–Schwarz inequality
x, y
2
x, x y , y
Two sides are equal if and only if they are linearly dependent (in geometry space, they are parallel) Expanding
Example(cont.)
• For Maximize Likelihood (ML), we have:
l ( ) log p( xi ) where 1 , 2 ,, k
N i 1
and the is the only parameter to be estimated, with the known type of pdf, the can be directly computed. However, for the previous case, the computing L is impossible since i l ( ) are expressed as follows:
*A.P.
Dempster, N.M. Laird, and D.B. Rubin, Maximum likelihood from incomplete data via the EM algorithm, J. Royal Statiscal Soc., Ser. R, vol. 39, no.1, pp.1-38, 1977.
n 2 xn 1 yn 1 xi yi xi yi i 1 i 1
n
2
2 2 n 1 n 1
2 xn 1 yn 1
2 n 1
n 2 n 2 n 2 n 2 xi yi xi yi i 1 i 1 i 1 i 1
EM Algorithm
Directory
• Theory • Simulation • Summary
Part I: Theory
Expectation Maximization
• Proposed by (A. P. Dempster, 1977)* • Motivation
Incomplete data hidden variable: in statistics, an extraneous variable in a statistical model that correlates (directly or inversely) with both the dependent variable and the independent variable The existing of hidden variable means solve problem directly based on formula is infeasible.