Deep learning driven blockwise moving object detection with binary scene modeling

Deep learning driven blockwise moving object detection

with binary scene modeling

Yaqing Zhang,Xi Li n,Zhongfei Zhang,Fei Wu,Liming Zhao

Zhejiang University,China

a r t i c l e i n f o

Article history:

Received26December2014

Received in revised form

22May2015

Accepted22May2015

Communicated by Shuiwang Ji

Available online2June2015

Keywords:

Moving object detection

Deep learning

Feature binarization

a b s t r a c t

As an important and challenging topic in computer vision,moving object detection is typically posed as a

problem of scene analysis,which usually requires both robust feature description and effective statistical

modeling.In general,the conventional detection methods make use of pre-de?ned hand-crafted features

and sophisticated background models for scene analysis.Therefore,they usually have a low general-

ization capability of adapting to different scenes with diverse spatio-temporal motion information.In the

face of high-de?nition video data,sophisticated statistical modeling often suffers from an expensive

computation or memory cost because of its low ef?ciency in evaluation and parallelization.In order to

address this issue,we propose a deep learning based block-wise scene analysis method equipped with a

binary spatio-temporal scene model.Based on the stacked denoising autoencoder,the deep learning

module of the proposed method aims to learn an effective deep image representation encoding the

intrinsic scene information,which leads to the robustness of feature description.Furthermore,the

proposed binary scene model captures the spatio-temporal scene distribution information in the

Hamming space,which ensures the high ef?ciency of moving object detection.Experimental results

on several datasets demonstrate the effectiveness and ef?ciency of the proposed method.

&2015Elsevier B.V.All rights reserved.

1.Introduction

Recent years have witnessed an increasing demand of effectively

understanding and compressing high-de?nition video surveillance

data in real-time.Typically,such a task is usually cast as a problem of

moving object detection[1],which aims to capture the object motion

information in different https://www.360docs.net/doc/d916708734.html,ually,moving object detection

is composed of two modules:feature representation and statistical

scene modeling.Speci?cally,the goal of feature representation is to

effectively re?ect the intrinsic structural properties of scene pixels,

while statistical scene modeling aims at building a robust scene

model that is capable of adaptively capturing the underlying spatio-

temporal distribution information on scene pixels.

In general,the traditional detection methods directly employ a set

of hand-crafted features for feature representation,which takes a

?xed and pre-de?ned feature extraction mode[2–8].For instance,the

commonly used color features(e.g.,raw pixel intensity[6])only

re?ect the visual perception properties of scene pixels,and often

ignore the local contextual information,resulting in the sensitivity to

noise and illumination changes.The texture features(e.g.,LBP[2])

encode the local geometric information on scene pixels,and are only

suitable for the particular textural scenes.The gradient-based features

(e.g.,HoG[5])are designed to capture the local shape properties of

scene pixels,and have a low generalization capability of re?ecting the

other properties of scene pixels.The transform coef?cient features

(e.g.,3D Walsh–Hadamard transform[8])map the scene pixels into a

set of orthogonal subspaces,and use the transform coef?cients as the

feature representation that only encodes some particular transform

information on scene pixels.Consequently,the aforementioned

features are incapable of well adapting to different types of scenes

with diverse appearance characteristics,resulting in a low general-

ization capacity for scene analysis.

Moreover,most existing methods for statistical scene modeling

often take a sophisticated modeling strategy for capturing the

spatio-temporal distribution characteristics of scene pixels,and

therefore suffer from a high computation or memory cost

[6,9–11,10,12–15].Typically,such modeling methods include mix-

ture of Gaussian model(MOG)[6],kernel density estimation(KDE)

model[9],and subspace learning using principal component analy-

sis(PCA)[10].In theory,these methods are often time-consuming

because of their frequent?oating operations with high memory

usage for high-resolution videos.To make scene modeling more

?exible,a number of sparse optimization based scene models

[12,13]are recently proposed;but they cannot be highly scalable

due to the high complexity induced by the?1optimization.In recent

Contents lists available at ScienceDirect

journal homepage:https://www.360docs.net/doc/d916708734.html,/locate/neucom

Neurocomputing

https://www.360docs.net/doc/d916708734.html,/10.1016/j.neucom.2015.05.082

0925-2312/&2015Elsevier B.V.All rights

reserved.

n Corresponding author.

E-mail address:xilizju@https://www.360docs.net/doc/d916708734.html,(X.Li).

Neurocomputing168(2015)454–463

years,deep learning is applied for dynamic background learning [16];however,this work does not consider the further transforma-tion for computational ef ?ciency.

In this paper,we propose a deep learning driven method for blockwise moving object detection based on binary scene model-ing.The proposed method constructs a deep feature representation by learning a stacked denoising autoencoder.After that,the learned feature representation is mapped into the binary Hamming space,and we construct a density analysis based scoring function for moving object detection.The main contributions of this work are summarized as follows:

1.We introduce deep learning to construct a robust feature representation for moving object detection.The learned feature representation is capable of well capturing the intrinsic struc-tural properties of a scene and adaptively discovering a set of ?lter patterns that are robust to complicated factors such as noise and illumination variation.Based on the deep learning framework,we further present a hash method to binarize the feature representation for substantially reducing memory usage in the process of dealing with long-duration high-resolution videos.The hash method is very ef ?cient in terms of several simple thresholding operations,which lays a solid foundation for real-time moving object detection.Note that the framework of feature binarization driven by deep learning for ef ?cient scene modeling is novel in the literature .

2.We propose a blockwise binary scene model that ef ?ciently encodes the spatio-temporal distribution information on the scene blocks.Such distribution information is measured by computing a density analysis based scoring function in the binary Hamming space,which ensures the computational ef ?ciency of moving object detection.

2.Our method 2.1.Overview

As shown in Fig.1,our method consists of two modules:

(1)stacked denoising autoencoder (SDAE)learning;and (2)binary scene modeling based on density analysis.More speci ?cally,we make use of deep learning to build an SDAE-based deep image representa-tion for encoding the intrinsic structural information for a scene.For

computational ef ?ciency,we further propose a hash method to generate binary codes for effectively preserving the statistical proper-ties of the learned feature representation in the Hamming space.Based on the hash method,we present a binary scene modeling method based on density analysis,which captures the spatio-temporal distribution information (measured by Hamming distance evaluation)for a scene in the binary Hamming space.Therefore,our method can not only learn a robust feature representation,but also construct a binary scene model based on density analysis,which results in the effectiveness and ef ?ciency of moving object detection.2.2.Deep feature learning by SDAE

Motivated by building an effective image representation for the robustness to complicated scene factors (e.g.,noise,illumination,and local variation),we take advantage of deep learning techniques [17,18]to learn the intrinsic high-level structural scene information from a huge amount of generic natural image patches (represented by gray-scale pixels),which are generated from the Tiny image dataset [19].Fig.2gives an illustration of our deep learning architecture for scene analysis.

As shown in Fig.2(a),a 3-layer denoising autoencoder network is constructed,including the input layer,the hidden layer after encoding,and the output layer after decoding.For robustness,the original input data is typically perturbed by adding additive Gaussian noise.Subsequently,the autoencoder tries to learn a

mapping function h W ;b e~x

i T%x i to approximate and recover the original signal by establishing the low-rank hidden layer units,which encode the intrinsic high-level structural information on image patches.In mathematics,the autoencoder network is asso-ciated with the following optimization problem:min

W ;b 12t X t i ?1

J x i à^x i J 22tλ2J W e1TJ 2F tJ W e2TJ 2F ;e1T

where

~x

i ?C ex i T;y i ?f W e1T~x i tb e1T

;^x i ?f W e2Ty i tb

e2T

:e2T

Here,eW e1T;W e2TTand eb e1T

;b e2T

Tdenote the weight and bias vari-ables for the encoder and decoder,respectively.C eáTis a function to

Parameters for feature extraction

Fig.1.The architecture of our method including the components of deep feature learning and binary scene modeling.

Y.Zhang et al./Neurocomputing 168(2015)454–463455

perturb x i .f eáTis an activation function (e.g.,logistic sigmoid function).λis a trade-off factor.t indicates the training data size.Some of the learned feature patterns by our autoencoder is visualized in Fig.3.

To further improve the power of feature representation,we adopt a stacked version of the denoising autoencoder [20],as shown in Fig.2(b).With the input image patch size being 16?16,the stacked denoising autoencoder sets up the layer-by-layer neural network by treating the activation output of the previous layer as the input of the next layer.In this way,we can symme-trically learn each encoding layer and its corresponding decoding layer one by one.Because of better capturing the data structure,a set of overcomplete ?lters are used as the input in the ?rst layer.As the network layer structure grows,the hidden unit size of the encoder is usually reduced by half.Conversely,the number of corresponding hidden units in the decoder will increase symme-trically.The stacked denoising autoencoder ends up with learning a low-dimensional latent representation which serves as the ?nal

feature representation for image patches.Furthermore,Hinton et al.[21]further re ?ned the structure of the stacked autoencoder by enforcing the sparsity constraints on the mean of activation values.Without loss of generality,let y i A R l denotes the output of the hidden encoder layer with the activation function being the

logistic sigmoid function,and ^ρA R l denotes the average activa-tion:^ρ

?1t X

t

i ?1y i

e3T

Moreover,a sparsity target ρis introduced to approximate the

average activation ^ρ

j ,where ^ρj denotes the j th element of ^ρ.To achieve this,the Kullback –Leibler (K –L )divergence can be intro-duced as an additional penalty term:

X l j ?1KL ρj j ^ρj ?àX l j ?1

ρlog ρ^ρj t1àρàálog 1àρ^ρj "#

:e4

T

Fig.2.Illustration of our deep feature learning scheme.(a)Shows the denoising autoencoder,and (b)displays the con ?guration of our stacked denoising autoencoder with the input patch size being 16?16.

Fig.3.Illustration of the learned feature patterns by our SDAE.

Y.Zhang et al./Neurocomputing 168(2015)454–463

456

Therefore,we have the ?nal optimization problem formulated as

min W ;b

J W 1eT;b 1eT;W 2eT;b 2eT;X ;^X

?min

W ;b 12t X t i ?1J x i à^x i J 22tλ2J W 1eTJ 2F tJ W 2eTJ 2F tβX l j ?1

KL ρj j ^ρj

e5T

where the parameter βcontrols the degree of sparsity penalty.The

whole network is ?ne-tuned using backpropagation algorithm with suf ?cient training data and time.The detailed procedure of the stacked autoencoder is described in Algorithm 1.Algorithm 1.Deep feature learning by

SDAE.

2.3.Binary scene modeling with density analysis

The ef ?ciency and computational cost are two basic concerns on

foreground detection in real-time applications.Inspired by hash-based methods for fast feature indexing,we propose the feature binarization method for ef ?cient scene modeling,which leads to reducing the computational cost signi ?cantly.The advantages are listed in the following two aspects.Firstly,with feature binariza-tion,we can perform bitwise operations on the binary data structures,such as Hamming distance,which is extremely fast and easy to parallelize.In addition,we can make full use of the memory because of the compact binary data structure,which allows us to build the application on high-resolution videos.Fig.4illustrates the framework of our scene modeling method.

Density analysis on binary features :Before describing our binary scene modeling method,we ?rst discuss about density analysis.Kernel density estimation (KDE)[9]is a non-parametric way to estimate the probability density function of a random variable.Its kernel density estimator

^f h ex T?1n X n i ?1K h

ex àx i Te6T

tries to estimate the unknown density by the independent and identically distributed samples x i i ?1;…;n ,where K h eTis a kernel function.Inspired by this work,we de ?ne a density analysis

based

Fig.4.An illustration of the scene modeling.The ?rst stage is to model the background parameters,Z el Tt 0;σel Tt 0are the statistics of the blockwise SDAE feature while Bg el T

t 0is the collection of the recent blockwise binary background features.In the testing stage,we extract the SDAE feature through a feedforward neural network.The feature is then

binarized to B el Tt 0by (11)using the background model.The model will compute the distance between the binary feature B el Tt 0and the background model Bg el T

t 0using Hamming

distance in (9)(as is shown in the ?gure,each row of Bg el T

t 0represents a binary vector of the background).Background classi ?cation will be done by thresholding in (10).

Y.Zhang et al./Neurocomputing 168(2015)454–463457

scoring function G (x ):G x eT?

1N X

N i ?1

D x ;x i eT;e7T

where D x ;x 0eTis a distance measure,and x i f g N i ?1denotes a collec-tion of observed feature vectors.To speed up the evaluation

process,the above scoring function is further reformulated in the Hamming space:D x ;x 0eT?

1k X

k j ?1x ej T

èx 0ej T;e8T

where x ej Tis the j th binary bit of x and èdenotes the xor operation.

For computational ef ?ciency,we binarize the feature represen-tation for each block.Let B el TA 0;1f g k ?n denotes the binarized feature vectors representing the block-speci ?c spatio-temporal cube with k being the number of binary bits.If each image block can be represented by a binary vector,the distribution for the

incoming binary feature B el T

r of the r th frame at the location l is estimated by using the corresponding q previous blocks

Bg el T

r ?B n i el Ti ?1;…;q such that S el T?0:

G B el Tr

?1q X q

i ?1

D B el Tr ;B el Tn i

?1kq X q

i ?1X k j ?1B el Tr ;ej T

èB el T

n i ;ej T :e9T

Then,we de ?ne the decision function for determining the mem-bership belonging to foreground or background:

S ex T?

0if G ex To η

1otherwise &

;e10Twhere ηis a threshold and G eáTis referred to in (9)and the block is determined as foreground when S ex T?1.

Feature binarization :The next key issue is how to perform feature binarization for time and memory ef ?ciency of density estimation in (9).The following describes the process of feature binarization based

on hashing.Let X el T

i be the input image block,which is represented by a spatio-temporal https://www.360docs.net/doc/d916708734.html,ing Algorithm 2,X el Ti is represented as a deep learning feature vector Y el Ti .Subsequently,each dimension of Y el Ti is binarized into 2bits by thresholding

B el Ti ;e2j à1;2j T?10if Y el Ti ;ej T4T el Tej T↑01if Y el Ti ;ej To T el Tej T↓

00otherwise

8>><

>>:

e11Twhere T el Tej T↑and T el T

ej T↓denote two threshold factors.In what follows,we describe how to adaptively determine the con ?gurations of these two threshold factors.We assume that the dimensions of the feature Y el Ti

are mutually uncorrelated,and follow Gaussian distributions.Mathematically,the j th feature element of the i th sample Y el T

i satis ?es

the following condition:Y el Ti ;ej T$N Z el Tej T;eσel Tej TT2

where Z el Tej Tand eσel Tej TT

2

denote the mean and variance,respectively.Based on the estimated

Z el Tej Tand eσel Tej TT2,the thresholds will be chosen as T el Tej T↑?Z el Tej Ttd σel Tej TT el Tej T↓?Z el Tej Tàd σel Tej T;

e12T

where d A R tis a factor determining the bandwidth of thresholds

(d A R tis empirically set to 5$10).

Model update :To allow for real-time applications,the online

update of the scene model for S el T?0is necessary.Since Y el T

i follows the Gaussian distribution,it is easy to know that the deviation Y el Ti ;ej TàY el Ti t1;ej T

also follows the Gaussian distribution N 0;2eσel Tej TT

2

.In this work,we mainly focus on the problem of stationary scene modeling.Therefore,the variance eσel Tej TT2

is usually

varies both gradually and slowly,and can be approximately assumed to be ?xed.As a result,the only factor for online updating

is the mean Z el Tj based on the previous observations f Y el Tp ;ej Tg such that

p ?i à1;i à2;…;i àq .

The above binarization method only provides one possible approach for binarization.Since our framework is ?exible,any other binarization (hash)methods can be used to improve the ef ?ciency of scene modeling.Algorithm 2summarizes the detailed work ?ow of our moving object detection method.

Algorithm 2.Binary scene modeling with density

analysis.

Table 1

Different scenes of videos.Scene type Video name Number of frames used Size of the frame Description

Baseline

Highway 1230320?240Highway with stable scene PETS2006730720?576Railway station with occlusion Pedestrians 629352?240Pedestrians in the park

Videos at night Streetcorner 1700592?240Street at night with illumination changes Tramstation 1200480?288Street at night with illumination changes Dynamic background Boats 1700320?240Boats in the lake with running water Canoe 719320?240Boats in the lake with running water Bad weather

Blizzard 1600720?480Heavy blizzard Wetsnow

1400

720?528

Heavy snow

Y.Zhang et al./Neurocomputing 168(2015)454–463

458

3.Experiments

3.1.Experimental setup

Dataset description:To demonstrate the effectiveness and ef?-ciency of our method,we conduct a number of experiments on nine challenging video sequences from https://www.360docs.net/doc/d916708734.html,[22] under four different types of scenes.These video sequences involve a set of complicated factors such as noisy perturbation,scene blurring induced by bad weather and illumination changes.In these Table1shows the detailed con?gurations of all the video sequences.Some typical frames in these videos will be chosen for our comparison.

Competing methods:In the experiments,we make a qualitative and quantitative comparison with four representative methods, including MoG,KDE,PBAS,and Texture.The MoG method adap-tively learns a mixture-of-Gaussian model to capture the statistical distribution information on scene pixels.In contrast,KDE tries to estimate the underlying scene distribution by building a kernel-based probabilistic density function.PBAS also models the scene by a history of recently observed pixel values but with dynamic controllers.Texture feature based scene modeling uses local binary patterns as local descriptors.These competing methods are imple-mented in a blockwise manner.

Evaluation criteria:We quantitatively evaluate the performance of the competing methods in the two aspects:accuracy and computational time.By considering moving object detection as a binary object classi?cation problem,we introduce the following two measures for accuracy performance evaluation:

Precision?

TP

TPtFP

;Recall?TP

TPtFN

;e13T

where TP denotes the number of correctly detected blocks(true positive),FP represents the number of incorrectly detected blocks (false positive),and FN stands for the number of incorrectly undetected blocks(false negative).Based on the these two

Table2

Architectures and con?gurations of our method.

Name Input block

size SDAE input

size

Con?guration of feedforward

SDAE

Number of scene training

frames

Spatio block

size

Binary feature

size

Background buffer

size

dη

Ours-

gray-

s216?1616?168?8256-512-256-128-64,64-128-

64-32-16

10025123050.06

Ours-rgb-

s1

16?16?316?16768-512-256-128-6410022563050.06

Ours-rgb-

s216?16?316?168?8768-512-256-128-64,192-128-

64-32-16

10025123050.06

Table3

F-measure comparison of processing on videos with different levels of noise.

Name Highway PETS2006Pedestrians StreetCorner TramStation Boats Canoe Blizzard WetSnow Average

(a)Experiments with no noise

MoG0.47280.52310.78440.60810.58840.78360.21500.27130.36290.5122 KDE0.43730.39160.67600.31510.10860.24190.18100.38020.26320.3328 Texture0.7609–0.31410.56170.44940.77070.51650.5811–0.5649 PBAS0.80390.64780.54140.56300.67410.80760.38050.72160.66900.6454 Ours-gray-s20.84830.70740.78190.59150.56930.50580.31490.78790.26650.5971 ours-Rgb-s10.87890.78280.78120.70790.75180.51590.69230.81560.54090.7186 Ours-rgb-s20.87800.79220.83520.73090.79320.65600.72580.85720.56700.7595

(b)Experiments with0.1Gaussian noise

MoG0.36750.45580.82150.52220.56730.65950.14960.26740.29940.4567 KDE0.20370.09860.58200.01760.04020.00720.09910.11490.03590.1332 Texture0.0455–0.42750.03540.01970.01390.01190.0010–0.0793 PBAS0.54760.64260.62540.39820.53470.22520.25530.17450.34910.4170 Ours-gray-s20.80500.56890.87470.26560.30840.25350.50080.38730.25730.4691 Ours-rgb-s10.85950.79070.82490.43950.70480.51660.75610.52100.40680.6467 Ours-rgb-s20.83680.81010.84330.42280.79320.63280.72650.51140.39180.6632

(c)Experiments with0.4Gaussian noise

MoG0.29610.49540.68420.43100.54390.35290.18510.33920.07260.3778 KDE0.48930.38580.70410.01150.04050.00400.35620.12310.02150.2373 Texture0.1612–0.73420.30000.20290.09800.32940.0100–0.2622 PBAS0.63600.63040.76380.29060.52700.38410.51950.29230.16600.4677 Ours-gray-s20.84080.65960.83400.48380.66400.36700.65610.62640.38060.6125 Ours-rgb-s10.84350.83650.75920.57780.72260.44260.83180.74880.36590.6810

Ours-rgb-s20.82540.82880.76160.53260.72770.54930.83250.77800.3555

0.6879

Fig.5.Visualization of F-measure comparisons.

Y.Zhang et al./Neurocomputing168(2015)454–463459

measures,we have a uni ?ed evaluation measure de ?ned as F àmeasure ?2?

precision árecall

precision trecall

:

e14T

The higher the value of F -measure,the better the detection performance.

Architectures of our method :The ?rst architectures we will use is the basic SDAE architecture as explained in Section 2.2.Each 16?16block of gray-scale pixels will be represented by a spatio-

temporal binary vector through multi-scale SDAE feature extrac-tion and binarization.Furthermore,the second architecture uses the blocks of RGB-scale pixels with the same SDAE structure as the ?rst one to convince the ef ?ciency of the RGB-scale input features.In addition,to fully illustrate the effect of multi-scale SDAE,we will use the architecture with a single-scale SDAE.Details of the three architectures of SDAE and the parameter con ?gurations are listed in Table 2,where d and ηdenote the thresholds referred to in (12)and (10).The parameters are tuned by hand to minimize

the

Fig.6.Illustration of the methods in resisting Gaussian noises with different variances.(a)Shows the perturbed version of the frame in the highway video sequence,(b)displays the groundtruth,(c)exhibits the results of our method for moving object detection,(d)shows the results of using mixture of Gaussian model,and (e)corresponds to the kernel density estimation

method.

Fig.7.Video surveillance captured with different scenes.

Y.Zhang et al./Neurocomputing 168(2015)454–463

460

https://www.360docs.net/doc/d916708734.html,parison:streetCornerAtNight with four methods (ours,KDE,PBAS,texture).The frame number of each row is 164,279,1114,1390from top to

bottom.

https://www.360docs.net/doc/d916708734.html,parison:boat with four methods (ours,KDE,PBAS,texture).The frame number of each row is 130,160,173,183from top to bottom.

Y.Zhang et al./Neurocomputing 168(2015)454–463461

F -measure over all of the datasets,and they are insensitive to different scenes.

Implementation :All models are trained and tested on MATLAB 2013b in an Intel i74770CPU (@3.40GHz)and a single NVIDIA Titan Z GPU (6GB RAM).3.2.Results

Overview :Table 3illustrates all the comparisons over different types of scenes under three levels of noisy corruptions by F -measure.Obviously,our method with the input of RGB-scale pixels outperforms all the other methods on the blockwise F -measure evaluation.Moreover,even the method based on gray-scale input can compete with state-of-the-art approaches (e.g.PBAS and texture-based methods)and performs more reliably on the highly perturbed videos.Fig.5visualizes the average F -measure performance on all the methods and test videos.

Experiments with noisy videos :Here,we aim to evaluate the effectiveness of our method in dealing with noisy perturbation.We perturb the video frames by adding different levels of noise.Fig.6shows the qualitative moving object detection results of our method against MoG,KDE and the accuracy comparison is shown in Table 3.Clearly,our method achieves visually more feasible detection results than the other methods.If the variance of additive Gaussian noise is above 0.1,all the other four methods perform poorly in moving object detection.In contrast,our method can still detect the moving objects with a relatively high accuracy.Even at the variance level of 0.40,our method can still succeed in ?nding out the main object.Fig.5visualizes the F -measure evaluation through different noise intensities.Obviously,our system remains a fairly stable performance compared with others.Therefore,the deep learning part of our method is capable of resisting the in ?uence of noisy perturbation.

Experiment with dynamic videos :In this experiment,we aim to evaluate the performance of our method against the competing methods in scene modeling with dynamic background.Fig.7shows the performances of our method in different scenes,compared with MoG,KDE,texture-based method [2]and pixel-based adaptive segmentation [23]and the accuracy comparison is shown in Table 3.Since our method utilizes deep learning to capture the intrinsic structural properties of a scene,the learned feature representation by our method is robust to the complicated factors such as illumination changes,bad weather with heavy snow,shadows and occlusions,etc.In contrast,MoG is quite sensitive to dynamic scenes,and KDE method has poor perfor-mance when the foreground and the background are similar.Texture features are often weak in handling the case of dynamic scene modeling (e.g.,for lakes)with high computational cost,while PBAS method usually fails to deal with illumination changes.In addition,we test our method on several other videos in Table 1and the system remains robust.In addition,more comparisons to verify the robustness and stability of our method are shown in Figs.8and 9.

Results of running time :In the cases of using different frame resolutions or block sizes,we quantitatively evaluate the perfor-mance of our method in the aspects of testing time.The

quantitative evaluation results are reported in Table 4.From Table 4,we observe that our method is able to achieve a very fast moving object detection performance with a moderate frame resolution or block size.Since our method extract blockwise multi-layer deep features to achieve better accuracy,it is reason-able to have a slower computing time compared to the methods using the raw pixels as the feature,such as KDE and MoG.

4.Conclusions

In this paper,we have proposed a highly effective and ef ?cient method for moving object detection based on binary scene modeling.The proposed method takes a stacked autoencoder-based deep learning scheme to adaptively build a robust feature representation,which is capable of well capturing the intrinsic structural properties of a scene and adaptively discovering a set of ?lter patterns that are robust to complicated factors such as noise and illumination variation.To improve the computational ef ?-ciency,we have developed a simple but effective hash method (by thresholding)to binarize the feature representation (obtained by deep learning)for substantially reducing memory usage in the process of dealing with long-duration high-resolution videos.Furthermore,we have presented a blockwise binary scene model that ef ?ciently encodes the spatio-temporal distribution in the formation on the scene blocks.Such distribution information is measured by computing a density analysis based scoring function in the binary Hamming space,which ensures the computational ef ?ciency of moving object detection.Experiments on several challenging video sequences have demonstrated the effectiveness of our method.Meanwhile,our method can be easily applied to real-time automated video analysis in different scenes due to extremely fast running speed and low memory usage (implemen-ted in GPU).

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 61472353,the China Knowledge Centre for Engineering Sciences and Technology,the Fundamental Research Funds for the Central Universities,the National Basic Research Program of China under Grant 2012CB316400,the NVIDIA CUDA Research Center Program and by the Zhejiang Provincial Engineering Center on Media Data Cloud Processing and Analysis.The work of Z.Zhang was also supported by the U.S.National Science Foundation under Grant CCF-1017828.References

[1]A.Yilmaz,O.Javed,M.Shah,Object tracking:a survey,ACM Comput.Surv.38

(4)(2006)13.

[2]M.Heikkila,M.Pietikainen,A texture-based method for modeling the back-ground and detecting moving objects,IEEE Trans.Pattern Anal.Mach.Intell.28(4)(2006)657–662.

[3]O.Javed,K.Sha ?que,M.Shah,A hierarchical approach to robust background

subtraction using color and gradient information,in:Workshop on Motion and Video Computing,2002,pp.22–27.

[4]P.Tang,L.Gao,Z.Liu,Salient moving object detection using stochastic

approach ?ltering,in:Fourth International Conference on Image and Graphics,2007,pp.530–535.

[5]S.Chen,J.Zhang,Y.Li,J.Zhang,A hierarchical model incorporating segmented

regions and pixel descriptors for video background subtraction,IEEE https://www.360docs.net/doc/d916708734.html,rm.8(1)(2012)118–127.

[6]C.Stauffer,W.E.L.Grimson,Adaptive background mixture models for real-time

tracking,in:IEEE Computer Society Conference on Computer Vision and Pattern Recognition,vol.2,1999.

[7]M.Harville,G.Gordon,J.Wood ?ll,Foreground segmentation using adaptive

mixture models in color and depth,in:Proceedings IEEE Workshop on Detection and Recognition of Events in Video,2001,pp.3–11.

Table 4

Performance of our method in running time and memory usage.Frame size Ours-gray-s2(ms)Ours-RGB-s2(ms)MoG

(ms)KDE (ms)Texture (ms)PBAS (ms)320?240 5.1

7.1 2.5 6.2125.023.0720?48021.025.212.011.1405.196.51280?96032.9

38.0

35.0

31.0

–

277.2

Y.Zhang et al./Neurocomputing 168(2015)454–463

462

[8]Y.Moshe,H.Hel-Or,Y.Hel-Or,Foreground detection using spatiotemporal

projection kernels,in:2012IEEE Conference on Computer Vision and Pattern Recognition (CVPR),2012,pp.3210–3217.

[9]A.Elgammal,R.Duraiswami,D.Harwood,L.S.Davis,Background and fore-ground modeling using nonparametric kernel density estimation for visual surveillance,Proc.IEEE 90(7)(2002)1151–1163.

[10]N.M.Oliver,B.Rosario,A.P.Pentland,A Bayesian computer vision system for

modeling human interactions,IEEE Trans.Pattern Anal.Mach.Intell.22(8)(2000)831–843.

[11]C.Guyon,T.Bouwmans, E.-H.Zahzah et al.,Robust principal component

analysis for background subtraction:systematic evaluation and comparative analysis.Principal Component Analysis,10,2012,pp.223–238.

[12]J.Huang,X.Huang,D.Metaxas,Learning with dynamic group sparsity,in:IEEE

12th International Conference on Computer Vision,2009,pp.64–71.

[13]Z.Cong,W.Xiaogang, C.Wai-Kuen,Background subtraction via robust

dictionary learning,EURASIP J.Image Video Process,vol.2011,special section p1.

[14]X.Zhou,C.Yang,W.Yu,Moving object detection by detecting contiguous

outliers in the low-rank representation,IEEE Trans.Pattern Anal.Mach.Intell.35(3)(2013)597–610.

[15]G.Zen,J.Krumm,N.Sebe, E.Horvitz, A.Kapoor,Nobody likes mondays:

foreground detection and behavioral patterns analysis in complex urban scenes,in:Proceedings of the 4th ACM/IEEE International Workshop on Analysis and Retrieval of Tracked Events and Motion in Imagery Stream,ACM,2013,pp.17–24.

[16]P.Xu,M.Ye,X.Li,Q.Liu,Y.Yang,J.Ding,Dynamic background learning through

deep auto-encoder networks,in:Proceedings of the ACM International Con-ference on Multimedia,ACM,2014,pp.107–116.

[17]I.J.Goodfellow,Q.V.Le,A.M.Saxe,H.Lee,A.Y.Ng,Measuring invariances in

deep networks,in:NIPS,vol.9,2009,pp.646–654.

[18]N.Wang,D.-Y.Yeung,Learning a deep compact image representation for visual

tracking,in:NIPS,2013,pp.809–817.

[19]A.Torralba,R.Fergus,W.T.Freeman,80million tiny images:a large data set for

nonparametric object and scene recognition,IEEE Trans.Pattern Anal.Mach.Intell.30(11)(2008)1958–1970.

[20]P.Vincent,https://www.360docs.net/doc/d916708734.html,rochelle,https://www.360docs.net/doc/d916708734.html,joie,Y.Bengio,P.-A.Manzagol,Stacked denoising

autoencoders:learning useful representations in a deep network with a local denoising criterion,J.Mach.Learn.Res.11(2010)3371–3408.

[21]G.Hinton,A practical guide to training restricted Boltzmann machines,

Momentum 9(1)(2010)926.

[22]N.Goyette,P.-M.Jodoin,F.Porikli,J.Konrad,P.Ishwar,https://www.360docs.net/doc/d916708734.html,:a

new change detection benchmark dataset,in:IEEE Computer Society Con-ference on Computer Vision and Pattern Recognition Workshops (CVPRW),2012,pp.1–8.

[23]M.Hofmann,P.Tiefenbacher,G.Rigoll,Background segmentation with feed-back:the pixel-based adaptive segmenter,in:IEEE Computer Society Con-ference on Computer Vision and Pattern Recognition Workshops (CVPRW),2012,pp.38–

43.

Yaqing Zhang is currently a second year Ph.D.student at the Department of Information Science and Electronic Engineering,Zhejiang University.He received the B.Sc.degree from Zhejiang University in 2013.His research interests include machine learning,computer vision and deep

learning.

Xi Li is currently a full professor at the Zhejiang University,China.Prior to that,he was a senior researcher at the University of Adelaide,Australia.From 2009to 2010,he worked as a postdoctoral researcher at CNRS Telecomd ParisTech,France.In 2009,he got the doctoral degree from National Laboratory of Pattern Recognition,Chinsese Academy of Sciences,Beijing,China.His research interests include visual tracking,motion analysis,face recognition,web data mining,image and video

retrieval.

Zhongfei Zhang is a professor at the Computer Science Department,State University of New York (SUNY)at Binghamton,USA,and the director of the Multimedia Research Laboratory in the Department.He received a B.S.in Electronics Engineering (with Honors),an M.S.in Information Sciences,both from Zhejiang University,China,and a Ph.D.in Computer Science from the University of Massachusetts at Amherst,USA.He was on the faculty of Computer Science and Engineering Department,and a research scientist at the Center of Excellence for Document Analysis and Recognition,both at SUNY Buffalo,before he joined the faculty of compu-ter science at SUNY

Binghamton.

Fei Wu received the B.Sc.degree from Lanzhou University,Lanzhou,China,in 1996,the M.Sc.degree from the University of Macau,Macau,China,in 1999,and the Ph.D.degree from Zhejiang University,Hang-zhou,China,in 2002,all in computer science.He is currently a full Professor with the College of Computer Science and Technology,Zhejiang University.His cur-rent research interests include multimedia retrieval,sparse representation,and machine

learning.

Liming Zhao is currently a second year Ph.D.student in College of Computer Science at Zhejiang University.His advisors are Prof.Xi Li and Prof.Yueting Zhuang.Earlier,he received his bachelor's degree from Shandong University in 2013.His current research interests are primarily in visual tracking,object detection and deep learning.

Y.Zhang et al./Neurocomputing 168(2015)454–463463