HMM based action recognition using Oriented histograms of optical flow field4202049[1]

HMM based action recognition using oriented histograms of optical?ow?eld

X.Li

A simple and effective motion descriptor based on oriented histograms

of optical?ow?eld sequence is proposed.After dimensional reduction

by principal component analysis,it is applicable to human action

recognition using the hidden Markov model(HMM).Experiments

with convincing recognition rate demonstrate the discriminative power

of the proposed descriptor.

Introduction:In recent years,human action recognition has become an active research area owing to its potential applications such as video surveillance,human–computer interface,content based retrieval and sports video analysis.Although many approaches for action recognition have been proposed,it is still a challenging task because of the non-rigid property of human movements and the ambiguous de?nition of different actions.Recently,the hidden Markov model (HMM)has been successfully applied to the task of human action recognition[1–5].Reference[1]represented human action as a sequence of movelet codewords.Reference[2]described action as a continuous sequence of discrete postures derived from an invariant descriptor.Reference[3]used the Cartesian component of optical ?ow velocity and the silhouette feature vector for action recognition. References[4]and[5]proposed coupled-HMM and primitive-based coupled-HMM to improve the performance of the action recognition https://www.360docs.net/doc/c15319726.html,ually the performance of previous methods for human action recognition relies on the success of pre-processing procedures, such as robust detecting and tracking of body parts[1]or accurate extraction of silhouettes[2,3].The tracking of body parts often fails due to occlusion or low quality of the video.If the foreground subject shares similar colour distribution with the background,the resulting extracted silhouettes might be imperfect.Also,the silhouette does not respect the relative motion of body parts inside the target region, which is important for action recognition.Unlike[4,5],which exploited different graphical model structures of HMM architecture, the emphasis of this Letter is on feature extraction and representation. The Letter proposes a simple and effective motion descriptor based on oriented histograms of optical?ow?eld sequence,without any silhouette extraction or tracking of body parts.After dimensional reduction by principal component analysis(PCA),the discriminative feature is applicable to human action recognition using the hidden Markov model.

Discriminative feature extraction:The histogram has been success-fully applied in many computer vision tasks,e.g.the colour histogram was used to compute the likelihood of the predicted region with the reference region in the particle?lter framework for robust object tracking[6].The histogram based SIFT descriptor,which can be constructed by?rst dividing the target region into non-overlapping grids and then computing the histograms of oriented gradients (HOGs)of each grid,has been used for robust object recognition [7].This Letter further extends the concept of HOGs from a still image to the optical?ow sequence for human action recognition.It is shown that for a speci?c action class,the dynamic oriented histograms of the optical?ow sequence along the temporal axis can represent the action discriminatively.This is reasonable because the dynamical patterns of oriented histograms of the optical?ow?eld,which describes the relative motion distributions of body parts,would be statistically similar for the same action class.The details are as follows.First,the sequence of optical?ow?elds is computed using standard approaches such as the Lucas-Kanade algorithm and we denote the optical?ow?eld for a speci?c time instance as G.Then the magnitude m(x;y)and the orientation y(x;y)of the optical gradients are computed by:

mex;yT?

????????????????????????????????????????

G xex;yT2tG yex;yT2

q

e1T

yex;yT?arctaneG yex;yT=G xex;yTTe2TAlthough the optical?ow corresponding to the background should be zeros since the actions are captured using a stationary camera,there are always noises.We extract the foreground movement via simple thresholding.It should be noted that,because the histogram is inher-ently a global statistical measure,no accurate silhouettes are needed here and the?nal recognition performance is insensitive to the selection of the valve.We quantise the orientation y(x;y)for all pixels with magnitude above threshold T into B orientation bins weighted by its corresponding magnitude m(x;y).Then the histogram H is normalised to ensure unit length such as:

P B

m?1

HemT?1e3T

Suppose there are C action classes and class i contains N i,i?1,...,C training sequences.Let H?i j2R B?l i j,i?1,...,C,j?1,...,N i be the j th motion histogram sequence for action i,where l i j denotes the sequence length of the j th training video for i th action class,the oriented histograms of optical?ow for the whole training set can be represented by H??[H?11,...,H?1N1,H?21,...,H?C1,...,H?C N c].The original histograms are B dimension,which is too high to be processed in the following procedure.We project H?to the linear subspace spanned by the?rst b principal components via PCA transform.The dimension of the projected histograms H i j2R b?l i j is much lower than the original one and it provides a more compact representation of the action dynamics. This discriminative feature will be used as input for HMM training in the following Section.

Action modelling and recognition using HMM:HMM has been successfully used for speech recognition and computer vision[8]. We employ the HMM for action modelling and recognition since it can be applied to model the time series data well.HMM allows for the inclusion of dynamics to model the action sequences.We built the HMM for different action classes using the learned discriminative features described in the preceding Section.The HMM model for the c th action class is given by l c?{A c,B c,p c}with N number of hidden states.Here A c is the transition matrix and p c is the initial distribution. The B c parameter represents the probability distributions for the observed feature vector conditional on the hidden states.Suppose each state has a bank of M Gaussian components,then the parameter B c consists of the following items:the mean vector m im,the mixture coef?cient c im and the full covariance matrix

P

im

for Gaussian component m in hidden state i,where m?1,...,M,i?1,...,N. The model parameters are learned in such a way that the likelihood p(O c j l c)is maximised by using the given set of training data set O c, which denotes the training feature sequence of the principal compo-nent analysis of oriented histograms of the optical?ow for action class c.The Baum-Welch algorithm[8]is used to iteratively re-estimate model parameters to achieve the local maximum.Given a test sequence for an unknown action with the corresponding feature vector sequence O,we use the maximum likelihood approach for the

classi?cation:

Fig.1Flowchart of approach

To represent motion patterns we?rst compute oriented histograms of optical?ow ?eld sequences.There are C action classes and each class i has N i training sequences.If length of j th sequence for action i is l i,j and denotes H?i j the j th motion histogram sequence for action i,then H?i j2R B?l i j.The principal component analysis is used to reduce the high dimension of H?i j into a lower one,i.e H i j2R b?l i j,where b<

ELECTRONICS LETTERS10th May2007Vol.43No.10

arg max c2f1;...;C g peO j l cTe4TThe?owchart of our approach is illustrated in Fig.1.Given a collection of labelled videos,our goal is to learn the HMM models for different classes of actions using the extracted discriminative motion features,i.e. the PCA of the oriented histograms of optical?ow?eld sequences,and apply the learned model to classify new video sequences.

Experimental results:Experiments were performed on image sequences of144?180pixel resolution and25frames per second. Nine representative action classes in real life are tested which include ‘walking’,‘running’,‘waving with one hand’,‘waving with two hands’,‘jumping forward’,‘jumping in place’,‘jumping in place with hand waving’,‘side-walking’and‘bending’.For simplicity,we denote those action classes as A1,A2,...,A9.Each action is performed by nine different people.The duration of each video is approximately four or?ve periodical cycles for the speci?c action performed.The dimension of the oriented histogram is reduced from original bins90to5via PCA transform.We use the HMM with the topology of six hidden states and each observation is modelled using mixtures of three Gaussian densities.In the experiments we performed ninefold cross-validation,i.e.for each action class,we randomly selected one sequence as testing data and the remaining eight sequences as training data.The procedure repeated20times and the average error across all trials is computed.Table1shows the confusion matrix of the action recognition experiment.The columns represent the ground truth and the rows represent the recognised actions.The recognition rates for A1,A2,...,and A9are100,85,85, 95,80,90,90,95and100%,respectively.The overall recognition rate is91.11%.Some sequences are misclassi?ed owing to the relative high degree of similarity between them,such as‘running’and ‘jumping forward’.

Table1:Confusion matrix for experiment(columns represent ground truth,rows represent recognised actions) A1A2A3A4A5A6A7A8A9

A12010010010

A20170030000

A30017100000

A40021900000

A50100160000

A60000018200

A70000021800

A80110000190

A90000000020

Rate100%85%85%95%80%90%90%95%100%Conclusion:This Letter extends the concept of the histogram of oriented gradients from still image to the optical?ow sequence. Experiments with satisfactory results show that the feature of the oriented histograms of optical?ow?eld sequence is discriminative for human action recognition.

Acknowledgment:The author thanks K.Fukui for valuable comments.

#The Institution of Engineering and Technology2007

19January2007

Electronics Letters online no:20070027

doi:10.1049/el:20070027

X.Li(Computer Vision Laboratory,Department of Computer Science,Graduate School of Systems and Information Engineering, University of Tsukuba,1-1-1Tennoudai,Tsukuba305-8573,Japan) E-mail:xili@viplab.is.tsukuba.ac.jp

References

1Feng,X.L.,and Perona,P.:‘Human action recognition by sequence of movelet codewords’.Proc.1st Int.Symp.on3D Data Processing Visualization and Transmission,2002,pp.717–721

2Kellokumpu,V.,Pietikainen,M.,and Heikkila,J.:‘Human activity recognition using sequences of postures’.Proc.IAPR Conf.on Machine Vision Applications,2005,pp.570–573

3Ahmad,M.,and Lee,S.W.:‘HMM-based human action recognition using multiview image sequences’.Proc.Int.Conf.on Pattern Recognition, 2006,pp.263–266

4Brand,M.,Oliver,N.,and Pentland,A.:‘Coupled hidden Markov models for complex action recognition’.Proc.Int.Conf.on Computer Vision and Pattern Recognition,1997,pp.994–999

5Ren,H.B.,and Xu,G.Y.:‘Human action recognition with primitive-based coupled-HMM’.Proc.Int.Conf.on Pattern Recognition,2002,pp.494–498

6Nummiaro,K.,Koller-Meier,E.,and Van Gool,L.:‘Object tracking with an adaptive color-based particle?lter’.Proc.24th DAGM Symp.on Pattern Recognition,2002,pp.353–360

7Lowe,D.G.:‘Distinctive image features from scale-invariant keypoints’, https://www.360docs.net/doc/c15319726.html,put.Vis.,2004,60,(2),pp.91–110

8Rabiner,L.R.:‘Tutorial on hidden Markov models and selected applications in speech recognition’,Proc.IEEE,1989,77,(2), pp.257–286

ELECTRONICS LETTERS10th May2007Vol.43No.10