Video Structure Analysis

TSINGHUA SCIENCE AND TECHNOLOGY

ISSN1007-021414/20pp714-718

Volume12,Number6,December2007

Video Structure Analysis*

ZHANG Songhai (张松海)1,**, ZHANG Yifei (张一飞)1, CHEN Tao (陈韬)1,

Peter M. Hall2, Ralph Martin3

1. Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China;

2. Department of Computer Science, University of Bath Bath BA2 7AY, United Kingdom;

3. School of Computer Science, Cardiff University, Wales CF24 3AA, United Kingdom

Abstract: Video structure analysis is a basic requirement for most content-based video editing and process-ing systems. This paper presents a fast video structure analysis method based on image segmentation in each frame, with region matching between frames. The structure analysis decomposes the video into sev-eral moving objects, including information about their colors, positions, shapes, movements, and lifetimes.

The method also supports user interactions to improve the results. The result shows that this method is fast and stable and can complete video analyzing interactively.

Key words: video structure analysis; video segmentation; image segmentation; region matching

Introduction

Video editing is widely used not only in professional movie making, but also in many other areas such as home videos, educational videos, and marketing videos. With the popularization of video cameras, even more potential applications are developing. Even relatively straightforward applications may wish to edit the video (for example, to remove an unwanted object from a scene) to process it in some other way (for example, to re-render it as a cartoon). In general, such tasks need an understanding of the structure of the video content, as a basis both for editing and for more advanced and complex video processing operations.

The need to treat the whole video in a coherent manner has been emphasized in several previous publications. For example, a video-tooning approach[1] pays particular attention to providing coherent segmen- tation in the space-time space to re-render the video as a cartoon animation. A recent paper on motion layer-based object removal in videos[2] also proposed a mo-tion-based video segmentation method as a preparatory step to object removal.

In many applications, this video analysis and seg-mentation step is the most time consuming step in the application. For example, video tooning may take sev-eral hours to segment a short video clip. This paper presents a fast video structure analysis method based on image segmentation in each frame, with region matching between frames.

1 Related Work

The two main kinds of information in any video are the color appearance and the visual motion, which leads to three approaches to video analysis. (1) The first is based on motion tracking[3] as in the method used in the motion layer-based object removal[2]. Such ap-proaches are suitable to videos with many highly tex-tured regions and rigid body motions, where trackers can readily follow moving objects and describe their

﹡﹡﹡Received: 2007-06-25

Supported by the National Key Basic Research and Development (973) Program of China (No. 2006CB303106), the Specialized Research Fund for the Doctoral Program of Higher Education of MOE, P.R.C. (No. 20060003057), and the Basic Research Foun-dation of Tsinghua National Laboratory for Information Science and Technology (TNList)

To whom correspondence should be addressed.

E-mail: zhangsh@https://www.360docs.net/doc/357454285.html,; Tel: 86-10-62797001-807

ZHANG Songhai (张松海) et al：Video Structure Analysis715

motion with a simple model. (2) Another approach is based on color and appearance. Such approaches are more suited to videos with relatively little texture and simple scenes. They can, to a certain extent, cope with various changes in objects, including deformation, splitting, and color changes. Such approaches can be further subdivided into space-time segmentation, such as performed by the anisotropic kernel mean shift algo-rithm[4], and segmentation in each frame separately fol-lowed by matching regions between frames, as in the stroke surfaces approach[5]. Space-time segmentation has the advantage that it identifies the entire object by observations throughout the video, which can, for ex-ample, help solve issues of occlusion, but such ap-proaches generally have long-run times and can be sensitive to the speed of the object’s motion and to variations in the object shape, color, and lighting, since this is a global approach. The stroke surfaces method is faster and can generally cope quite well with objects whose appearance changes, but its object matching ability needs to improve. The method described here follows a similar scheme of image segmentation in each frame, with region matching between frames. The greedy algorithm quickly matches regions between frames quickly, and combines motion tracking and simple user interaction to improve the results. (3) The third approach is based on both color and motion in-formation. Such approaches often introduce a statis-tics-based model[6-8] and use an expectation maximum (EM) algorithm to solve the optimization problem. Such approaches lose the efficiency and the conven-ience for user interaction, and are not suitable to fast video structure analysis processing.

2 Method

The video structure analysis algorithm combines image segmentation and region matching. The algorithm first segments each frame into regions based on the color properties. Image segmentation has been extensively studied[9-11], and any appropriate methods can be used in this system. However no segmentation algorithm is a universal method suitable for all kinds of videos, and each one has different efficiency characteristics and applicability. Thus, the segmentation algorithm should be appropriate to the nature of the input video. Of the well-known algorithms, tests show that mean shift segmentation[10] works well on videos with rigid bodies and motions as in Fig. 1, whereas histogram-based methods, i.e., color quantization[11], work well for videos containing non-rigid motion, for example

moving liquids, fire, and clouds as shown in Fig. 2.

Fig. 1

Mean shift segmentation

Fig. 2 Histogram-based segmentation

Even when using the most suitable algorithm for a given video, the segmentation result for each frame may typically have hundreds of regions, which is still a very complex representation. A higher level result needs to merge these regions in an appropriate way and to find associations between regions in adjacent frames of the video. Regions belonging to the same object are connected to derive a three-dimensional object in space-time. A given region may, for example, occluded by some other object in any given frame, in which case the region is split in that frame with the space-time representation branching at that frame. Conversely, parts of a region may merge when the occlusion disap-pears, so the object branches also merge in the space-

time description.

Fig. 3 Video structure graph

The graph

I O

(,,)

G V E E shown in Fig. 3 is defined to represent the video structure. The node set V is com-posed of all the regions in all frames resulting from the segmentation. E I is an inner edge set that connects

Tsinghua Science and Technology , December 2007, 12(6): 714-718

716 nodes within a given frame, which means that two re-gions within the frame belong to the same object. Cer-tain inner edges may be placed by the user, since hu-man intelligence may be able to detect meaningful connections which are very difficult for an algorithm to deduce. O E is an outer edge set that connects nodes between neighboring frames, denoting corresponding regions between frames. A region may have no corre-sponding regions in a previous frame, meaning that the region has come into existence in that frame. Similarly, a region may have no corresponding regions in a sub-sequent frame, meaning that the region vanishes in that frame. More generally, as well as the obvious 1-1 cor-respondences between regions in adjacent frames, 1-N correspondences will also occur with multiple outer edges connecting regions in the next frame, as objects splitting or as occluded. N -1 correspondences with multiple outer edges between regions in the previous frame represent object merging or no longer being oc-cluded. To simplify the problem, the algorithm as-sumes that there are no N -N correspondences in the graph.

The problem is represented as a minimization prob-lem by adding a virtual node representing a NULL re-gion in each frame. Any regions in the next frame which come into existence are connected by a virtual edge to this NULL region, and any regions which dis-appear in the previous frame are again connected by a virtual edge to the NULL region. In this way, apart from the NULL regions, every region has at least one edge connecting it to some region in the next frame, and at least one edge connecting it to some region in the previous frame.

A weight is associated with each edge to represent the similarities between regions. Larger weights indi-cate greater similarity between two regions. Virtual edges corresponding to 0-1 and 1-0 relationships have

a threshold weight w T , which is a user set parameter.

I E and O E determine the video structure and each connected sub-graph represents an object. If I E and O E are NULL, each region has become an isolated

object which only appears in one frame. The optimal structure of I E and O E can be formulated as a maximization problem: O

max ,e e E w ∈Σ subject to the

constraint that no N -N correspondences are formed be-tween regions, where e means the edge in O .E (Note that considering the connectivity of nodes in pairs of successive frames is not sufficient to ensure that no N -N correspondences between regions exist, as edges in previous or later frames may eventually join these nodes into a single region).

3 Edge Weighting Function

The edge weight measures the similarity between re-gions. The edge weighting function is set to negative infinity if the regions are too different (to prevent these regions from merging) and is set to the weighted sum of the weights of the three components:

d c 123,

spatialDis or colorDis ;areaRatio spatialDis colorDis.E w w w ???∞>>??=?

?×?×?×??areaRatio is the logarithm of the ratio of the areas of the two regions. colorDis is the distance in the CIE-Lab color space between the average pixel colors. spa-tialDis measures the spatial distance between two re-gions which is approximated by ellipses. The represent of an ellipse E is composed of c as the center, a and b as two lengths of axes, and θ as the angle between the long axis and x axis. And the distance of two ellipses E 1 and E 2 is shown as below, p is a point. spatialDis = D (E 1, E 2)=D 1 (E 1, C 2) + D 1 (E 2, C 1

).

1(,)D E p =

The weights and thresholds are chosen according to

the characteristics of the specific video. Often 10.8w =, 20.6w =, 30.2w =. If objects move rapidly

in the video, d ? should be larger, typically 30. If

lighting changes or the given objects change color, c ? should be larger, typically 10. 4 Correspondence The optimization algorithm can not be solved exactly

since the problem is intractable; therefore, a greedy al-gorithm was used to determine the region connectivity structure. The problem is solved frame-by-frame to

ZHANG Songhai (张松海) et al ：Video Structure Analysis 717

determine correspondences between successive frames, starting from the first frame. In each frame, the objec-tive is to find as many correspondences as possible while observing the constraint that there should be no N -N correspondences. In practice, the algorithm em-phasizes forming 1-1 correspondences. Details of the correspondence computation for frame i are shown be-low. The k -th region in frame i is .k i R

Step 1 Construct the new region set for frame i . If 1i =, i.e. the first frame, the new region set is simply set to the segmentation result. For other frames, search for any regions which connect to the same region in the previous frame, and merge these into a new bigger region. Then recompute its area, average color, and modeling ellipse. Also combine any regions for which the user has defined an edge in I E .

Step 2 Find 1-1 correspondences. A 1-1 corre-spondence is regarded as being the most stable and plausible relationship. Such relationships generally have a large weight, representing regions of similar color and size that move slowly without occlusions. Compute each region in frame i and each region in frame 1i +. If for any region in frame i , the edge be-tween k i R and 1l i R + has the maximum weight and at

the same time, for any region in frame 1i +, the edge between k i R and 1l i R + also has the maximum weight, then construct a 1-1 correspondence and add it into O E . Then mark these edges as used and repeat the process for the remaining edges until no more 1-1 cor-respondences of this kind can be found. Note that such 1-1 correspondences may be changed to 1-N or N -1 correspondences in latter steps. For example, a big re-gion could have edges with a region having a similar area and several small regions in the next frame. Step 3 Find 1-N and N -1 correspondences. Sort the remaining edges in descending order of weight. Then consider each edge in turn and add the edge to O E if it satisfies two constraints. First, adding the edge must not create an N -N correspondence. Second, the area difference between the one big region and the sum of the areas of the N small regions should be no more than 10%, since objects remain a similar size from frame to frame, even if they fragment.

Step 4 Attach isolated regions in frame i and frame 1i +. Join any isolated regions with neighbor- ing regions within the frame if they are sufficiently

similar in color and position. Any remaining isolated

regions in frame i are assumed to have disappeared in the next frame, while any remaining isolated regions in frame 1i + are assumed to be newly born objects. After Step 1, the user can add region correspon-dences into O E by hand, or a motion tracking algo-rithm can be used to determine the desired correspon-dences. Such edges are added and marked before pro-ceeding to Step 2.

After finding the correspondences between all suc-cessive pairs of frames, each connected sub graph in the final result corresponds to a single object. The in-formation about this object is then derived from the sub graph, including its shape, color, and behavior over time.

5 Results

The method was tested using the bearwavi.avi video, courtesy of Peter Hall of the University of Bath, using a computer with a 2.4-GHz CPU and 512 MB memory. The EDISON mean shift algorithm [9] was used for the image segmentation. The segmentation speed depends on the parameters used. If the color bandwidth and spatial bandwidth are set to 4, the segmentation takes only a few seconds for each frame and produces more than 1000 regions per frame. The algorithm then takes about 8 s to matching regions in 30 frames. Setting the color bandwidth and spatial bandwidth to 14 gives bet-ter segmentation results for the automatic region matching as shown in Fig. 4, but it then takes more than 60 s to segment each frame, producing about 80 regions per frame. The region matching was then done in less than one second. Several tests showed that the overall success rate for automatic region matching is about 68%.

The matching process was then extended from the stroke surfaces method, which can only handle some ideal cases. The method gives much better run time performance than video tooning. As the algorithm processes each frame, the user can interactively calcu-late the video structure. The algorithm continues image segmentation on the next frame while the user interac-tively improves the region matching on the current frame. The two processes will not conflict on a dual core CPU.

Tsinghua Science and Technology , December 2007, 12(6): 714-718

718

(a) Result image (b) Segmentation result (c) Automatic region matching result

Fig. 4 Results of segmentation and region matching. Top row: results for frame 1; Bottom row: results for frame 5.

6 Conclusions and Future Work

This paper describes a fast video structure analysis method based on image segmentation and region matching. The structure analysis decomposes the video into several moving objects, giving information about the color, position, shape, movement, and life time of each object. This method also supports user interaction to improve the results; the interaction needed is straightforward and obvious. Any image segmentation algorithm can be used in the first step to suit the source video characteristics. A fast greedy algorithm is used to approximately optimize the structure graph of the im-age segmentation results.

One minor problem is that the algorithm may cause objects to become too large in the last few frames. Al-ternative optimization methods which simultaneously consider correspondences in all frames may improve the results, but these would be slower and the improvement might not be worth the extra time spent. References

[1] Wang J, Xu Y, Shum H Y, et al. Video tooning. ACM

Transactions on Graphics , 2004, 23(3): 574-583. [2] Zhang Y, Xiao J, Shah M. Motion layer based object re-moval in videos. In: Proceedings of the Seventh IEEE Workshops on Application of Computer Vision (WACV/MOTION’05). Washington DC, USA: IEEE Computer Society, 2005: 1: 516-521.

[3] Shi J, Tomasi C. Good features to track. In: Proceedings of

IEEE Conference on Computer Vision and Pattern Recog-nition (CVPR’94). Seattle, USA, 1994: 593-600. [4] Wang J, Thiesson B, Xu Y, et al. Image and video seg-mentation by anisotropic kernel mean shift. In: Proceed-ings of European Conference on Computer Vision. Prague, Czech, 2004: 238-249.

[5] Collomosse J P, Rowntree D, Hall P M. Stroke surfaces:

Temporally coherent artistic animations from video. IEEE Transactions on Visualization and Computer Graphics , 2005, 11(5): 540-549.

[6] Chang M M, Tekalp A M, Sezan M I. Simultaneous mo-tion estimation and segmentation. IEEE Trans. on Image Processing , 1997, 6(9): 1326-1333.

[7] Zitnick C L, Jojic N, Kang S B. Consistent segmentation

for optical flow estimation. In: Proceedings of the Tenth IEEE International Conference on Computer Vision. Washington DC, USA: IEEE Computer Society, 2005: 2: 1308-1315.

[8] Khan S, Shah M. Object based segmentation of video us-ing color motion and spatial information. In: Proceedings of IEEE Conference on Computer Vision and Pattern Rec-ognition. Hawaii, USA, 2001: 2: 746-751.

[9] Felzenszwalb P F, Huttenlocher D P. Efficient graph-based

image segmentation. International Journal of Computer Vision , 2004, 59(2): 167-181.

[10] Comanicu D, Meer P. Mean shift: A robust approach to-ward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2002, 24(5): 603-619. [11] James Arvo. Graphics Gems II. San Diego, USA: Aca-demic Press, 1991.