A new maximum likelihood gradient algorithm for on-line hidden Markov model identification

A NEW MAXIMUM LIKELIHOOD GRADIENT ALGORITHM FOR ON-LINE HIDDENMARKOV MODEL IDENTIFICATIONIain B.CollingsDept.of Elec.&Electr.Engineering, University of Melbourne,AustraliaTobias Ryd´e nDept.of Mathematical Statistics, Lund University,SwedenABSTRACTThis paper presents a new algorithm for on-line identifi-cation of hidden Markov model(HMM)parameters.The scheme is gradient based,and provides parameter estimates which recursively maximise the likelihood function.It is therefore a recursive maximum likelihood(RML)algorithm, and it has optimal asymptotic properties.The only current on-line HMM identification algorithm with anything other than suboptimal rate of convergence is based on a predic-tion error(PE)cost function.As well as presenting a new algorithm,this paper also highlights and explains a counter-intuitive convergence problem for the current recursive PE (RPE)algorithm,when operating in low noise conditions. Importantly,this problem does not exist for the new RML algorithm.Simulation studies demonstrate the superior per-formance of the new algorithm,compared to current tech-niques.1.INTRODUCTIONRecently,hidden Markov models(HMMs)have been used ina wide variety of applications,including speech processing[8],frequency tracking[10]and signal estimation in mobile communication systems[2].In each of these areas,the related tasks of state estimation for known models and on-line model identification are of critical importance.On-line identification algorithms are usually based on either recursive maximum likelihood(RML)or recursive predic-tion error(RPE)techniques.In the context of HMMs,two slightly different RML schemes are proposed in[5]and[6], and an RPE scheme is given in[1].The difference between the RML schemes essentially concerns the scaling matrix that pre-multiplies the gradient of the log-likelihood;in[6] this matrix is derived using ideas of the EM(expectation-maximization)algorithm,while a different approach is taken in[5].This difference is not unimportant,however,as thethe constrained transition probability estimates.In Section6 simulation examples are given.2.PROBLEM FORMULATION2.1.State space signal modelLet be a discrete-time,homogeneous,first order Markov chain taking values in a set with elements.Without loss of generality,we can be identify these elements with the set of unit vectors,wherewith1in the position. (The prime denotes transpose.)It is well known that the dynamics of can be written as follows[3]:(1) where is the transition probability matrix with elements,and is a martingale increment.Of course andfor each.Also,consider the observation process(2) where without loss of generality,since is in afinite set(where denotes the inner prod-uct),and is the vector of state values of the Markov chain.In this paper we consider that is a white(or more precisely,independent and identically dis-tributed)Gaussian noise(WGN)process,with zero mean and standard deviation.Let be the probability density function of conditional on,and write.Thus,in our setting,(3) 2.2.Model parametrisationWe are concerned with recursive identification of the param-eters and;we assume to be known.It is clear that the set of valid rows in a transition probability matrix constitute a simplex.There is,however,an alternative parametrisa-tion,on a sphere,that is superior to the one on a simplex,as observed in[1].Specifically,let.Each row in the corresponding matrix belongs to the manifold(4) The parameter of interest can then be written(5)Clearly,both and depend on this parameter,that is and.The dimension of is ,representing state values and transition probabilities.The important aspect of the work in this pa-per is that the identification scheme involves decoupling which greatly reduces the computational complexity.In cases where the parameter is not as in(5),the framework described below can still be used,with appropriate changes to the gradient expressions which follow.3.PARAMETRISED INFORMATION-STATESIGNAL MODELLet the information-state,,be the vector of conditional probabilities under,defined as follows:(6) This vector may be recursively updated using the following equation:diag(9) and(10)withdiag(11) Also,,and it should be pointed out that the derivatives are taken in the direction perpendicular to the constraint surface(4).Now,by collecting the elements of and into one column vector vec we obtain the recursive updatevec(12)4.THE NEW RML ALGORITHMThe log-likelihood can be re-written as(13) where denotes a log-likelihood increment.Thus,the conditional score function(not given here due to space limitations)can be written in terms of,, and,given in(8)to(11).We can now construct our new RML algorithm as follows:(14)where is a projection onto the constraint domain,vec and,True values are 0and 1Figure 1:Level estimates of 2state Markov chain,True value is 0.9Figure 2:Transition probability estimates of 2state Markov chain(both algorithms converge)Figure 3:Error function for levelestimates(both algorithms converge)Figure 4:Error function for transition probability estimates7.REFERENCES[1]I.B.Collings,V .Krishnamurthy,and J.B.Moore,“On-line identification of hidden Markov models via recur-sive prediction error techniques”,IEEE Trans.Signal Process.,vol.42,pp.3535–3539,1994.[2]I.B.Collings and J.B.Moore,“An HMM approachto adaptive demodulation of QAM signals in fading channels,”Int.J.Adapt.Control Signal Process.,vol.8,pp.457–474,1994.[3]R.J.Elliott,“Exact adaptive filters for Markov chainsobserved in Gaussian noise,”Automatica ,vol.30,no.9,1976.[4]V .Fabian,“On asymptotic normality in stochastic ap-proximation,”Ann.Math.Statist.,vol.39,pp.1327–1332,1968.[5]U.Holst and G.Lindgren,“Recursive estimation inmixture models with Markov regime,”IEEE rm.Theory ,vol.37,pp.1683–1690,1991.[6]V .Krishnamurthy and J.B.Moore,“On-line estima-tion of hidden Markov model parameters based on the Kullback-Leibler information measure,”IEEE Trans.Signal Process.,vol.41,pp.2557–2573,1993.[7]L.Ljung and T.S¨o derstr¨o m,Theory and Practice ofRecursive Identification .Cambridge,MA:MIT Press,1983.[8]L.R.Rabiner,“A tutorial on hidden Markov modelsand selected applications in speech recognition,”Proc.IEEE ,vol.77,pp.257–285,1989.[9]D.Ruppert,“Almost sure approximations to theRobbins-Monro and Kiefer-Wolfowitz processes with dependent noise,”Ann.Prob.,vol.10,pp.178–187,1982.[10]R.L.Streit and R.Barrett,“Frequency line trackingusing hidden Markov models,”IEEE Trans.Acoust.Speech Signal Process.,vol.38,pp.586–598,1990.。