Soccer Ball Tracking using Dynamic Kalman Filter with Velocity Control

Soccer Ball Tracking using Dynamic Kalman Filter with Velocity Control

Jong-Yun Kim, Tae-Yong Kim

GSAIM, Chung-Ang Univ South Korea

{ idsrge@https://www.360docs.net/doc/606907116.html,, kimty@cau.ac.kr }

Abstract

In this paper, we propose the ball tracking method that is tracking the ball adaptively and robustly in the soccer video. In the latest works, people have used the Typical Kalman Filter to track the ball. But when the ball is disappearing due to the occlusion with players, Typical Kalman Filter has no choice but to make a pool prediction and especially if the player take the ball for a long time, the error is produced much more. To overcome these problems, we propose the Dynamic Kalman Filter algorithm. Dynamic Kalman Filter robustly tracks a ball in the dynamic condition by using player information and reduces the error in the situation of occlusion by controlling the velocity of the state vector. The experimental results show that proposed Dynamic Kalman Filter shows better results than the Typical Kalman Filter and the Adaptive Kalman Filter that is proposed to overcome occlusion problem in the video sequence.

Keywords--- Dynamic Kalman Filter, Ball Tracking, Velocity Control

1. Introduction

In the past few years, the tracking object in the video sequence is very important issue. Especially, analysis and tracking the moving object in the sports video is one of the most popular researching parts in many kinds of video sequence because we use these works for the analysis of tactics and of player's moving pattern in the games and the soccer is most famous sports in the world, so tracking object in the soccer video becomes more meaningful work than any other sports videos.

Targets of tracking in the soccer video are the player and the ball mostly. Tracking the movement of the player is very important researching part because it can be applied to know the strategy of attacking and defense of teams. Tracking the player in the soccer video has been studied in the [1]-[4] and they have used the color, shape, location of player as information of tracking. Tracking a ball in the soccer video is also very important because it can be applied to know the ratio of sharing the ball and which team has superiority in the game and also we can track the viewer's ROI (Region of Interest) part by chasing a ball position in the soccer video. But it is very difficult to detect and track the ball in the soccer video because the size of the ball is too small and it has only a little amount of features. So many researchers study the method to solve these problems recently.

In the past time, people used the feature of appearance of the object, seems like a color, shape, size, texture to track the ball in the video. T. D'Orazio used the information of color of ball and [7] Xiao-Feng Tong detected the ball using feature of circularity of object [6] and D. Liang used the weighted sum of feature information for evaluation value to detect the ball in the soccer video [8] and K. Seo used the feature of appearance of a ball and he thought that the ball must be tracked on the longest time in the soccer video. So he tracked the ball using these features [5]. But there are many objects that are similar in shape with the ball in the soccer video, for example line, socks of players, and distortion of an image, so it is not robust method using the simple feature of appearance of the ball in the single frame. Therefore other researchers used not only features of appearance but also trajectory information in the consecutive frames. One of the most used methods to track the ball trajectory is Kalman Filter algorithm which is very famous method in the object tracking, and used in the [9]-[11] to track a ball in the soccer video. In the [9], X. Yu tracked a ball using Kalman Filter-based Template matching algorithm and chased trajectory of ball candidates and selected the best trajectory by using some evaluation methods [10], [11]. This method shows a better result than previous methods because it is comparatively robust to track the ball in the bad condition when that appearance of a ball is changed during several consecutive frames. But Typical Kalman Filter couldn't get a good prediction or miss the target when a ball is occluded with the player or field line. Then this bad prediction also gives negative effects to the next prediction. Therefore the error is accumulated frame by frame and Kalman prediction is not corrected. So we need to get more robust method to track the ball in the dynamic condition in the soccer video. Hence in this paper, we propose a Dynamic Kalman Filter algorithm for tracking a ball more robustly.

2009 Sixth International Conference on Computer Graphics, Imaging and Visualization

2. The Typical Kalman Filter

First we investigate the problems of ball tracking

using the Typical Kalman Filter. In the Typical Kalman

Filter, the system state and measurement model are

defined in simple equations as follows.

1

2

In the equations, A is a transition matrix, H is a

measurement matrix to connect between state and measurement. The random variable and are the process and measurement noise respectively. They are

assumed to be independent each other and covariance is

Q and R, which have zero average and white Gaussian

probability distribution. When we track the ball using Kalman Filter in the soccer video, the position and velocity are defined in the state model and Kalman Filter performs the tracking by using two step algorithm, "Prediction" and "Correction".

The first step of the Typical Kalman Filter is

"Prediction". In the first step, we can get the predicted state and error covariance as follows.

3

4

The Second step is "Correction". In this step, The Typical Kalman Filter calculates the Kalman Gain using the previously predicted error covariance (5). Then the Kalman Filter corrects the state model by Kalman gain and measurement value that is the result of measurement in the sequence (6). Then Finally Kalman Filter corrects the error covariance using Kalman Gain (7) and turns over these values, corrected state and error covariance to the prediction step again.

5

6

– 7

Through this recursive algorithm, we can track a ball in the soccer video. The Block Diagram of the Typical Kalman Filter is presented in Figure 1.

Figure 1 Block Diagram of Typical Kalman Filter

If we can measure the ball during the long periods in soccer video, the Typical Kalman Filter tracks a ball very well. But in the bad situation, something like a ball occluded with a player or other object in the video sequence, it is very difficult to detect and track a ball very well. Figure 2 presents the bad situation for tracking by Typical Kalman Filter.

(a) (b) (c)

Figure 2 Ball Tracking Result using Typical

Kalman Filter

Green mark of x is the prediction point and red mark of x is the measurement point. In the Figure 2 (a), the Typical Kalman Filter tracks the ball very well but in the Figure 2 (b)-(c), the ball is occluded with player, therefore Kalman Filter does not detect a ball and Kalman prediction maintains the previous velocity and direction. Hence the prediction becomes to have more distance with ball position more and more as time goes by. This is the problem what we must overcome when we want to track a ball robustly using Kalman Filter. 3. Dynamic Kalman Filter with Velocity Control

3. 1. Overview of Our Algorithm

Figure 3 Overview of Our Algorithm

Figure 3 presents the overview of our algorithm. We can classify view of the Soccer video by three types: global view, middle view and close-up view. The middle and close-up view are not appropriate to track a ball because these views only show the player, coach, spectator with closed-up shot, so the ball is not showed for a long time. Therefore we only track a ball in the global view because this shot shows a ball more long times and the global view makes up the majority of the sequence of the soccer video. Sometimes a ball is out of the soccer field in the global view but this case is very difficult to detect and track the ball, so we assume that we track the moving ball that is only in the ground. The entire object that we want to find (players and the ball) is in the ground, so first we detect the area of the ground in the full-frame .

3.2. Player and Ball Candidate Detection

3.2.1. Ground Detection

In the global view, the biggest part of the frame is ground area, so dominant color of the frame is the ground color. Many researchers detect the ground using the dominant color detection in the global view. [1]- [8], but it is very difficult to know what amount of range is the ground color. If we set the color range broadly, non-ground object will be detected as the ground, and on the contrary, if we set the small color range, the ground pixel that has a color out of range will not be detected as the ground. Hence this method is not robust. So instead of these method, we propose the new method to evaluate which pixel is ground or not. In the [5], they defined the ground color using feature of ground color pixel and HSV color space. They mentioned that ground color pixel has a G>R>B feature. G, R, B are the Green, Red, Blue in the RGB space respectively. Even if the entire ground pixel doesn't have the same color, the majority ground pixels has the same feature that is G>R>B . But this feature often appears in the gray-color pixels, the line, goalpost, socks of the player and they are recognized as the ground pixel. So we use the G>R>B feature and one more feature to detect the ground pixel. Our additional method is using edge information. Even though gray-color pixel has a G>R>B feature, they also have many edges. But ground pixels rarely have edge elements. So we use these mixed methods to detect the ground color as follows.

,

1 , , ,

, 0 ?

8

In the experiments, we used canny edge detection [15] to discriminate whether the pixel has an edge or not. Figure 4 shows the ground detection result using our method. Figure 4-(a) shows the result using only ground

color feature. This shows that ball is recognized as ground because a ball has a gray-color feature. Figure 4-(b) shows the result using only edge information and Figure 4-(c) shows the final result using the ground color and edge information. Then Figure 4-(d) shows the object discrimination result using the CCL (Connected Component Labeling).

(a)

(b)

(c)

(d)

Figure 4 (a) Color Detection (b) Edge Detection (c) Ground Detection Result (d) Labeling Result

3.2.2. Needless Objects Elimination

Because we only use the information of the player and ball to track the soccer ball, the seats (for the spectators), scoreboard, line and goalpost are the needless object to us. So these kinds of objects need to

be eliminated. We can eliminate these needless objects

that have a salient feature opposite from players and a ball. First the seats and scoreboard are much bigger than the player in the global view and their position is almost same in the consecutive frames. So we can easily eliminate these objects by using that information. But lines in the ground are difficult to eliminate. Although separated lines are easy to eliminate relatively, but it is very difficult for us to eliminate when the lines are occluded with the player and labeling one object. To overcome this problem, first we detect these occluded objects among the whole objects. These occluded objects have several salient features, first, the size of object is much bigger than the player, and second, the ratio of object pixel on the least square area pixel is low. Because these objects are composed by the player, line and large amount of ground pixels in the least square area. So if we use these salient features, we can separate these from the other objects. Now then what we thought is how to eliminate the line in these objects. The salient feature between the player and line is the density of the pixels. In the frame image, the line's shape is thin and long but on the contrary, player has thick, short shape and more density of pixel than line. So we first find the most density area in the object and eliminate the pixels except this area. In addition to this, if the player's uniform color is not gray-color, we can eliminate the line in the object by eliminating the pixels that have a G>R>B color feature. Figure 5 shows the result that we have eliminated the needless object using our proposed method.

Figure 5 Needless Objects Elimination Result

3.2.3. Player and Ball Candidate Separation

We can regard remained objects as the candidate of the player and the ball because the needless object was eliminated. Then we separate the player and ball candidates and use their information to track a ball.

In the global view, the player's height is longer than width because most of the player stands on the ground.

So we can define this feature as :

2. and are

width and height of the player respectively. Then also in the global view, we can define the average player size as : _ /16 _ _ /4. In this expression, _ and _ are height of the frame and player respectively.

In the global view, a ball has a feature that the ball's width and height are similar each other. So we can define

this feature as :

2

0.5. The official soccer

ball's diameter is about 9inch and when we assume that

average player height is about 71inch, we define this

feature as : _ /10 _ _ /5. In

this expression, player_h and ball_h are height of the

player and ball respectively.

Figure 6 shows the result of separation between the

player and ball candidates. Blue color square and red one

means the player and the ball candidates respectively.

Mostly the separation results are appropriate but we can

find the bad result in this figure that the referee’s leg is

classified as a ball candidate. But we can overcome these

kinds of errors in the tracking part of our algorithm.

Figure 6 Player and Ball Separation Result

3.3. Ball Tracking using Dynamic Kalman Filter

3.3.1. Motion Modeling

We will track a ball by using the results of the

separation and the Dynamic Kalman Filter. So first, we

need to make the model of the state and parameters in the

Kalman Filter as follows.

11

12

13

11

01

00

00

00

00

11

01

14

1000

0010

15

Where x k is the state model of the ball and ,

are x, y coordinates of the state. v x,v y are the velocity of

x, y-direction. The velocity is defined as the difference of

x, y coordinates between the current frame and previous

frame (12), (13). Accordingly, as the state is defined like

that, the transition matrix A and the measurement matrix

H are also defined as equation (14), (15).

3.3.2. Dynamic Kalman Filtering with Velocity

Control

With the motion modeling we have to design the

Dynamic Kalman Filter that is available to track a ball in

the dynamic conditions. Figure 7 and 8 shows our

proposed algorithm.

Figure 7 Block Diagram of Dynamic Kalman

Filter

In the Figure 7, the part shown in the dotted line is a difference with the Typical Kalman Filter in Figure 1. The Typical Kalman Filter tracks the target object using the Q (process noise covariance), R (measurement noise covariance) which are initialized as constant values and does not be changed until the tracking is over. It cannot track a ball properly in the dynamic condition, for example, a ball is occluded with players frequently. Therefore in this paper, we propose the Dynamic Kalman Filter that can track a ball in the dynamic condition by automatically changing the parameters. Figure 8 presents the three modes that are corresponded with three possible conditions in the soccer game.

Figure 8 Block Diagram of Dynamic Parameter

Setting

First, we have to set up the parameters Q, R, SA and Target according to the mode. In Figure 8, q and r are the diagonal element of matrix Q, R that is defined as process noise covariance and measurement noise covariance as below.

00

00

00

00

,

00

00

00

00

16

Matrix Q and R are important parameters to control the Kalman gain. In our proposed model, the Kalman gain is determined as the weight of position and velocity. At this time, if we define the weight of position is W and weight of velocity is W , they are concerned in Q, R as follows.

,

17

Hence we can do the dynamic tracking by controlling these parameters q, r with the mode appropriately. Next, we have to set up the parameter SA (Search Area). SA is classified as three kinds of mode - small, player, large. These respective modes can be represented by the distance from the target point – small = 20, large = 30 and especially the player = (DPLA/2) + 20 where the unit of calculation is pixel. DPLA means diagonal length of the player area. Lastly we have to set up the parameter Target. Target means the object that we have to track and that is classified as the ball and player mode.

Measurement Mode : this case means that we find the ball in the SA. At this time, we can trust the measurement value so we can set up the parameter r = 0 in order to apply the measurement entirely. Then we set up the q = 0. Then the Target mode is the ball because the ball is measured in this case.

Player Occlusion Mode : if we does not find a ball but the player is detected in the SA, we can assume that a ball is occluded with the player. In this mode, even if we do not find the ball, we can trust that the ball exists in the player area. So we can set up the parameter q = 1, r = 0 in order to reflect the measurement entirely. Instead, the Target is not a ball but the player and we have made the Kalman Filter to track the player until we find a ball around the player area.

Prediction Mode : if we do not find a ball and also the player does not be detected in the SA, we can assume that we cannot find a ball temporarily due to the noise of the frame. Hence in this case we cannot trust the measurement. so we can set up the parameter as q = 0, r = ∞ in order that the Kalman Filter tracks a ball using only the prediction value. Instead, SA is large because the measurement error is big and then in order that the Kalman Filter searches a ball in the large area.

Velocity Control : when we track a ball using the Kalman Filter, we need to control the velocity of the state vector before a ball is occluded with the player. When a ball is in the situation of Figure 9, if the ball does not pass through the player, we know that the player catches the ball during the several frames. So if we do not reduce the velocity of the state vector before the occlusion, the prediction is not stopped well like a Figure 9 because the prediction is affected from the velocity of

the state. This causes the negative results such that

Kalman prediction has a big error and missing the Target at in the worst case.

Figure 9 Velocity Control Problem To solve this problem, we need to control the velocity of the state. The Target is already changed in the Player Occlusion Mode and in the Prediction Mode, the Kalman Filter does not refer to the velocity of state, these modes are not related with the velocity control. Therefore we control the velocity of prediction in the Measurement Mode. In this mode, if we expect that the prediction point will be beyond the player area like a Figure 9, we can control the velocity of state vector by revising Prediction point like a Figure 10.

Figure 10 Velocity Control

means Predicted Position of the Ball, is Current Position of a ball and is Center Point of the Player. As we have previously mentioned, if we expect that the prediction point will be beyond the player area, we change the to by reducing the Velocity of the state. If we present that the is ( , ) and the is ( , ), we can denote according to the Kalman Filter expression. Thus , can be presented like a previous Figure 10 and also we can control the Velocity value using Velocity Control parameter ‘ ’ as below.

, , ,

,

(18)

19

4. Experimental Results

We have implemented the experimental environment using OpenCV ver1.0 [15]. Then we used the 592x320 soccer video that is the game from the 08-

09 English Premier League Round 8 Chelsea vs Middlesbrough and we used the consecutive 1000 frames

of global view in this video. We tested our algorithm with the two kinds of methods (Typical and Adaptive Kalman Filter) to compare and demonstrate the superiority of our algorithm. Especially we used the Weng’s method as Adaptive Kalman Filter algorithm [14]. This algorithm robustly tracks the target object in the situation when the target object is occluded with the other one in the video sequence. We used this algorithm and applied this to track a ball in the soccer video to compare with our algorithm. Figure 11 shows the result.

(a)

(b)

(c)

(d)

Figure 11 (a) Trajectory of Real Ball (b) Typical Kalman Filter (c) Adaptive Kalman Filter (d)

Dynamic Kalman Filter

In this Figure, the horizontal axis means the frame number and the vertical axis means the distance of measurement position from the origin. (a) shows the result of real ball position in the video sequence and (b), (c) and (d) show the tracking result using Typical, Adaptive and proposed Dynamic Kalman Filter. When we see the result of (b), Typical Kalman Filter does not track the ball well in the whole sequence because when a ball is occluded with the player, Kalman prediction has a big error and these errors are accumulated continuously.

(c) shows the better result than (b) because the Adaptive Kalman Filter tracks the ball robustly when a ball is occluded and passed through the player. But when a ball is occluded and the player possesses it during the several frames, Kalman prediction predicts the wrong position and big error can be occurred. However our proposed Dynamic Kalman Filter (d) shows good tracking result in the almost whole sequence and especially it overcomes the occlusion problem well. A little error was produced during the frame 720-800, because in this sequence, the players crowded in the middle of ground and when the Kalman Filter tracks a ball and chooses the wrong player as the target in the Player Occlusion Mode. But the proposed Dynamic Kalman Filter shows the robust tracking results in the almost whole sequence except these short frames. Figure12 shows the errors between the real ball position and the tracking results that we use the each Kalman Filter method.

(a)

(b)

(c)

Figure 12 (a) Difference of RB-TK (b) RB-AK (c)

RB-DK

In the Figure 12, RB means Real Ball and TK, AK, DK means Typical, Adaptive, Dynamic Kalman filter respectively and the results show that the proposed Dynamic Kalman Filter is the best robust ball tracking algorithm among them in our experiment. Table 1 shows the average of errors and processing time. In this table, performance time of the proposed Dynamic Kalman Filter is larger than others about 2.1% because of complexity of the algorithm. But an error is much smaller than others about 85%, 75% to compare with TK and AK respectively. So we can know DK is the best among them.

Algorithm Average of Error

(pixel)

Average of

Processing Time

(msec/frame)

TK 142.82 166.72

AK 83.95 166.89 DK 20.73 170.62 Table 1 Average of Error and Performance Time In the soccer video, if the prediction is close to real

ball position, we can tell that the prediction is correct and Kalman Filter has well prediction in this frame. Hence

we can assume that if the difference between the prediction and real ball position is smaller than threshold,

this means that the Kalman Filter has correct prediction

and detects a ball in this frame correctly. Table 2 shows

the accuracy of the algorithms. First rows of the table

show the threshold values. The results show that DK has

a high accuracy more than 81% when the threshold is

30pixel and it is 10pixel, DK is also more than 66 % accuracy. Table 2 also shows that DK’s accuracy is higher than TK and AK 83%, 38% respectively. Hence

we can say that our proposed DK algorithm tracks a ball robustly in the soccer video.

Algorithm Threshold(pixel)

10 20 30

TK 35.6% 41.5% 44.7% AK 48.4% 60.3% 66.0% DK 66.8% 78.5% 81.7% Table 2 Accuracy of Algorithms Conclusions

In this paper, we propose the new Dynamic Kalman Filter model to track the ball robustly by overcoming the player-ball occlusion problem in the soccer video sequence. First we classified three kinds of modes and

we have made Dynamic Kalman Filter that could set up

the parameters adaptively to accurately track a ball by conditions. We also have made the Dynamic Kalman Filter that could control the weighting value of the velocity to overcome the prediction error problem and to reduce the prediction error in the situation of player-ball occlusion. The results present that the Dynamic Kalman Filter robustly tracks a ball in the long sequence of frame better than the Typical Kalman Filter and Adaptive Kalman Filter. But as we previously mentioned, when

we meet the situation that the players are crowded in the ground, the proposed Dynamic Kalman Filter shows the negative result. Hence our future work is solving this problem to increase the accuracy of tracking a ball in the soccer video sequence.

Acknowledgements

This work was supported by the ITRC (Information Technology Research Center, MIC) and Seoul R&BD program, Korea.

References

[1] J. Liu, X. F. Tong, et al. Automatic Player detection,

labeling and tracking in broadcast soccer video. In

BMVC 2007.

[2] A. Ekin, A.M. Tekalp, and R. Mehrotra. Automatic

Soccer Video Analysis and Summarization, IEEE Trans.

on Image Processing, Vol. 12:7(2003), 796--807.

[3] V. Pallavi, J. Mukherjee, A.K. Majumdar and S. Sural,

Graph-based multiplayer detection and tracking in

broadcast soccer videos, IEEE Transactions on

Multimedia 10 (5) (2008).

[4] P. Figueroa, N. Leite, R. Barros, I. Cohen, and G.

Medioni. Tracking soccer players using the graph

representation. In ICPR, pages 787–790, 2004. [5] K. Seo, J. Ko, I. Ahn, and C. Kim, "An intelligent

display scheme of soccer video on mobile devices,"

IEEE Trans. Circuits Syst. Video Technol., vol. 17, no.

10, pp. 1395-1401, Oct. 2007

[6] T. D'Orazio , N. Ancona , G. Cicirelli , M. Nitti, A Ball

Detection Algorithm for Real Soccer Image Sequences,

Proceedings of the 16 th International Conference on

Pattern Recognition (ICPR'02) Volume 1, p.10210,

August 11-15, 2002

[7] Xiao-Feng Tong , Han-Qing Lu , Qing-Shan Liu, An

Effective and Fast Soccer Ball Detection and Tracking

Method, Proceedings of the Pattern Recognition, 17th

International Conference on (ICPR'04) Volume 4, p.795-

798, August 23-26, 2004

[8] D. Liang, Y. Liu, Q. Huang, and W. Gao, "A scheme for

ball detection and tracking in broadcast soccer video," in

Proc. Pacific-Rim Conf. Multimedia, 2005, pp. 864--875.

[9] X. Yu, H. W. Leong, C. Xu, and Q. Tian. Trajectory-

based ball detection and tracking of broadcast soccer

video, IEEE Transactions onMultimedia, vol 8, Issue 6,

pp: 1164-1178, December, 2006.

[10] X. Yu, C. Xu, Q. Tian, and H. W. Leong, “A Ball

Tracking Framework for Broadcast Soccer Video”,

ICME, Baltimore, US, July 6-9, 2003

[11] X. Yu, Q. Tian, and K. W. Wan. A novel ball detection

framework for real soccer video, In Proc. ICME 2003,

Vol II, 265--268.

[12] G.Welch and G. Bishop. An introduction to the kalman

filter. Technical Report 95-041, University of North

Carolina, Department of Computer Science, 1995. [13] Yi Zhang, Huosheng Hu, and Huiyu Zhou. Study on

Adaptive Kalman Filtering Algorithms in Human Movement Tracking. In Int. Conf. on Information

Acquisition. IEEE, 2005.

[14] Weng, S. K., Kuo, C. M., Tu, S. K., Video object

tracking using adaptive Kalman filter, Journal of Visual

Communication and Image Representation, Vol. 17, No.

6, 2006, pp. 1190-1208.

[15] “OpenCV library,” https://www.360docs.net/doc/606907116.html,/projects/open

cvlibrary.