Video Annotation Based on Kernel Linear Neighborhood Propagation

620IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 10, NO. 4, JUNE 2008Video Annotation Based on Kernel Linear Neighborhood PropagationJinhui Tang, Student Member, IEEE, Xian-Sheng Hua, Member, IEEE, Guo-Jun Qi, Yan Song, and Xiuqing WuAbstract—The insufficiency of labeled training data for representing the distribution of the entire dataset is a major obstacle in automatic semantic annotation of large-scale video database. Semi-supervised learning algorithms, which attempt to learn from both labeled and unlabeled data, are promising to solve this problem. In this paper, a novel graph-based semi-supervised learning method named kernel linear neighborhood propagation (KLNP) is proposed and applied to video annotation. This approach combines the consistency assumption, which is the basic assumption in semi-supervised learning, and the local linear embedding (LLE) method in a nonlinear kernel-mapped space. KLNP improves a recently proposed method linear neighborhood propagation (LNP) by tackling the limitation of its local linear assumption on the distribution of semantics. Experiments conducted on the TRECVID data set demonstrate that this approach outperforms other popular graph-based semi-supervised learning methods for video semantic annotation. Index Terms—Kernel method, label propagation, semi-supervised learning, video annotation.I. INTRODUCTIONAUTOMATIC annotation (also called high-level feature extraction in TRECVID benchmark [1]) of video and video segments is an elementary step for semantic-level video search. As manually annotating large video archive is labor-intensive and time-consuming, many automatic approaches are proposed to handle this issue. Generally, these methods build statistical models from manually pre-labeled samples, and then assign the labels for the unlabeled ones using these models. However, this process has a major obstacle: frequently the labeled data is insufficient so that the distribution of the labeled data does not well represent the distribution of the entire dataset (include labeled and unlabeled data), which usually leads to inaccurate annotation results. Over the recent years, the availability of large data collections, which only have limited human annotation, has attracted the attention of a growing community of researchers to theManuscript received September 15, 2007; revised December 30, 2007; accepted February 19, 2008. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Ling Guan. J. Tang, Y. Song, and X. Wu are with the Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei 230027, China (e-mail: jhtang@; songy@ustc. ; wuxq@). X.-S. Hua is with the Microsoft Research Asia, Beijing 100080, China (e-mail: xshua@). G.-J. Qi is with the Department of Automation, University of Science and Technology of China, Hefei 230027, China (e-mail: qgj@). Color versions of one or more of the figures in this paper are available online at . Digital Object Identifier 10.1109/TMM.2008.921853problem of semi-supervised learning [2]. By leveraging unlabeled data with certain assumptions, semi-supervised learning methods are promising to deal with the above obstacle. Many works on this topic are reported in the literature of machine learning community and some of them have been applied to video or image annotation. In [3], co-training is applied to video annotation based on a careful split of visual features. Yan et al. [4] analyze the drawbacks of co-training in video annotation, and propose an improved co-training style algorithm named semi-supervised cross-feature learning. A structure-sensitive anisotropic manifold ranking method is proposed in [5] for video concept detection, where the authors analyze the graph-based semi-supervised learning methods from the view of PDE-based diffusion. In [6], manifold ranking (a graph-based semi-supervised learning method) based on feature selection is proposed for video annotation. Wang et al. [7] propose a method based on random walk with restarts to refine the results of image annotation. Tang et al. [8] embed the temporal consistency of video data into the graph based semi-supervised learning and propose a temporally consistent Gaussian random field method for video annotation. The key point of semi-supervised learning is the consistency assumption [9]: nearby samples in the feature space or samples on the same structure (also referred to as a cluster or a manifold) are likely to have the same label. This assumption considers both the local smoothness and the structure smoothness of the semantic distribution. There are close relations between the consistency assumption and nonlinear dimensionality reduction schemes [10]–[12] since intrinsically they follow the same idea of reducing the global coordinate system of the dataset to a lower-dimensional one while preserving the local distribution structure. Recently, a method called linear neighborhood propagation (LNP) [13], [14] is proposed to combine these two strategies. LNP borrows the basic assumption of local linear embedding (LLE) [10], [12] that each sample can be reconstructed by its neighboring samples linearly, and further assumes that the label of the sample can be reconstructed by the labels of its neighboring samples using the same coefficients. This method potentially assumes that the mapping from the feature to label is linear in a local area (to be detailed in Section II). Usually the local linear assumption is reasonable such as in LLE. However, using this assumption to construct the propagation coefficients in graph-based semi-supervised learning has limited depictive ability for the real-world datasets. And to ensure the convergence of the iterative label propagation in graph-based semi-supervised learning, the reconstruction coefficients are required to be non-negative. This makes the liner label reconstruction not so perfect. If the semantic super-plane has a high curvature in this area, LNP will fail. In other words, if the labels of the samples1520-9210/$25.00 © 2008 IEEETANG et al.: VIDEO ANNOTATION BASED ON KLNP621Define the mapping from the feature to label as we can obtain,(2)Fig. 1. Roadmap of KLNP.and in the local area distribute complexly in the feature space, this linear assumption is not appropriate. Therefore this method is not suitable to tackle the video semantic annotation problem for some certain semantic concepts, since typically some semantic distributions of video segments, which are collected from different sources and span a large time interval, are very complex in the feature space. Motivated by the great success of kernel trick [15], we propose a novel method for automatic video annotation named kernel linear neighborhood propagation (KLNP), which also combines the consistency assumption and LLE but is applied in a nonlinear kernel-mapped space. Fig. 1 shows the roadmap of KLNP. This method is able to handle more complex situation since it holds both the advantages of LNP and kernel methods, through mapping the input feature space to a nonlinear kernel-mapped feature space. Compared to the previous work in [16], we reconstruct the label reconstruction coefficients in a more effective way: one more constraint is added into the optimization problem for coefficients reconstruction, that is, every coefficient should be non-negative. This constraint is necessary to ensure the convergence of the iterative label propagation process. The coefficients in [16] cannot ensure the convergence of label propagation, so it is hardly to implement as it needs to compute the inversions of large matrices for label prediction. The experiments conducted on the TRECVID [1] dataset demonstrate that KLNP is more appropriate than LNP for complex applications and can obtain a more accurate result than other semi-supervised learning methods for video high-level feature extraction. The rest of this paper is organized as follows. In Section II, we briefly introduce LNP and analyze its limitation; and the proposed KLNP for the video semantic annotation problem is detailed in Section III. We analyze the incremental extension of KLNP and show that KLNP is equivalent to KPCA plus LNP in Section IV. Experiments are introduced in Section V, followed by the concluding remarks in Section VI. II. LNP AND ITS LIMITATION In this section, we briefly introduce LNP and analyze its limitation. LNP is based on the assumption that the label of each sample can be reconstructed linearly by its neighbors’ labels, and the coefficients of reconstructing the label is the same as the ones for reconstructing the feature vector of the sample. This can be formulated as (1) where of . is the neighbors of sample , and is the label (3) Combine (2) and (3), we have (4) Equation (4) indicates that the mapping from the feature to label is linear in a local area. In many times, the local linear assumption is reasonable for feature vector reconstruction, such as in LLE [10]. However, using this assumption to construct the propagation coefficients in graph-based semi-supervised learning has limited depictive ability for the real-world datasets. In fact, the linear assumption is a reduced case of the nonlinear one. So by extending to nonlinear propagation, the depiction ability of the model is strengthened. One of contributions of this paper is to reveal that it is an important property to describe the complex real-world problems with this strengthened nonlinear depiction, such as real-world video/image annotation on News dataset. Although the linear assumption has gained success on some benchmark dataset, such as UCI, the performances on the real-world sets have only reported limitedly. From the experiments, we can see on real-world datasets, the nonlinear assumption outperforms the linear one significantly. Besides, to ensure the convergence of the iterative label propagation process in LNP [13], the reconstruction coefficients are required to be non-negative. This makes the linear label reconstruction not so perfect, such as using linear assumption with non-negative coefficients cannot reconstruct the upmost point well by its near neighbors in a two dimensional space but through nonlinear mapping we can reconstruct it better. Especially when the semantics distribute complexly, using local linear assumption with non-negative coefficients on the semantics to reconstruct the label is not very effective. For example, the average label reconstruction error1 of Building on the training data of TRECVID05 [1] is 0.236, which is a little high. And from the viewpoint of LLE, LNP assumes that the semantic is a 1-D manifold embedded in the feature space. Obviously 1-D manifold cannot model complex semantic distribution well. So this method cannot well tackle the video semantic annotation problem since some semantics of video segments often have very complex distributions. We call this drawback limitation of local linear assumption on the distribution of semantics. III. KERNEL LINEAR NEIGHBORHOOD PROPAGATION To tackle the aforementioned limitation of LNP, in KLNP, we map the features to a higher dimensional space through kernel1Average label reconstruction error: (1=n) is the reconstructed label of x .((f0 f^ ) =f^ ), where f622IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 10, NO. 4, JUNE 2008Fig. 2. Exemplary thumbnails of the 39 concepts.mapping and then try to obtain the reconstruction coefficients in this nonlinear space. This is motivated by the great success of kernel trick in pattern recognition area. KLNP also assumes that the label of each sample can be reconstructed linearly by its neighbors’ labels, but the label reconstruction in KLNP is more reasonable, as the kernel mapping is nonlinear and we can obtained better reconstruction coefficients in the kernel-mapped space. be a set of samLet (feature space ples (i.e., video shots for our application) in , where the sample with -dimensional features). set contains the first samples labeled for every concept, and as are unlabeled ones. Here the label “1” represents the sample is relevant for a certain concept while “0” represents the is irrelevant for the certain concept. It is well known that directly optimizing the integer label 1/0 is a NP hard problem. So these labels are usually relaxed to continuous labels (between 0 and 1). These continue label can be seen as the relevance score of each sample for a certain concept. And according to the task of high-level feature extraction in TRECVID [1], the objective here is to rank the remaining unlabeled samples, so the real-value scores are naturally suitable for this task. The vector of the predicted labels of all samples is represented as , which can be split into two blocks after the th row (5) Considering a kernel mapping space to a mapped space operating from inputThedatakernel matrixmapped . Then, of dot products can be represented assetcanbeto the(6) . For the kernel function, several where typical kernel functions, such as Radial Basis Function (RBF), Gaussian and sigmoid kernel, can be employed. In our experiments, we adopt the RBF as the kernel. of every We find the nearest neighbors using the following distance (for concision, we use to substi): tute(7) Please note this formula shows that the distance can be obtained directly from the kernel matrix instead of the mapping function . According to the assumption of LLE [10], can be linearly reconstructed from its neighbors. Using kernel mapping, the coefficients obtained in the mapped space can reconstruct the labels better than the coefficients obtained in the original feature space. To compute the optimal reconstruction coefficients, the reconstruction error of is defined as(8)TANG et al.: VIDEO ANNOTATION BASED ON KLNP623TABLE I COMPARISONS OF RESULTS BY USING THE FEATURE COLOR MOMENTSTABLE II COMPARISONS OF THE MAPS BY USING THE FEATURE COLOR MOMENTSwhereis the reconstruction coefficient for from , is the vector of is the entry of the “local” reconstruction coefficients, and of in the kernel-mapped space Gram matrix(10) where is a matrix formed by the mapped feature vectors for the nearest neighbors of th sample, “ ” is a -dimensional column vector with each entry equals to 1, and . Please note here the subscripts and do not mean is the element in th row and th column in in is decided by the positions the matrix. The position of and in . For example, if is the th column and of is the th column in , is of the element in th row and th column of . To assure the convergence of iterative label propagation, another constraint is added to reconstruct the propagation coefficients (i.e. reconstruction coefficients), which is the reconstruction coefficient should be non-negative. Then we can obtain the by solving the stanoptimal reconstruction coefficients for dard quadratic programming problem(11) Similar to the distance measure (see (7)), we can see that the Gram matrix in (10) also can be calculated directly from the kernel matrix instead of the mapping function. Therefore, in the entire computing procedure, the mapping function actually is not explicitly required. It is intuitive that the obtained reconstruction coefficients reflect the intrinsic local semantic structure of the samples in the mapped space. These coefficients will be applied to reconstruct the unlabeled samples’ labels (which are real values instead of “0” or “1”), that is, to estimate the prediction function . In order to obtain the optimal , we define the following cost functionEnforce the constraint that can be rewritten as, the reconstruction error(9) (12)624IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 10, NO. 4, JUNE 2008where is the label of sample . Minimizing this cost will optimally reconstruct the labels of all unlabeled samples from the counterparts of their neighbors. And from the view of label propagation, minimizing this cost results in iterative label information propagations from labeled samples to other samples according to the linear neighborhood structure in the nonlinear mapped space. Define a sparse matrix with the entry in the th row and th column3:4: 5: 6:(13) then this optimization objective is represented formally asin a matrix with th row including neighbors’ indices of th sample, also store their distances in a matrix ; according to (10), then Compute the Gram matrix can be computed by solving the optimization problem in (11); Construct the label propagation coefficients matrix according to (13); Iterate (18) until convergence, then the unlabeled samples’ real-value labels can be obtained; Rank the unlabeled samples according the predicted labels. IV. DISCUSSION AND EXTENSION(14) This is a standard graph-based semi-supervised learning problem [17]. Notice that and split the matrix after the th row and th column(15) we can represent the optimization problem in a matrix manner as(16) Similar to [17], the optimal solution can be obtained as (17) Since and , as discussed in [5], the specand are both less than 1, so the label tral radiuses of prediction can also be accomplished in an iterative manner, that is, iterate(18) until convergence. This is actually an information propagation process from the label of each sample to its neighbors. The main procedure of the above algorithm is summarized in Algorithm 1. Algorithm 1 KLNPSince multimedia database is always expanding quickly. So how to handle the incremental samples is an important topic in multimedia research [18]. Here we discuss this issue in KLNP. Assume that we have incremental samples (lawill be beled and unlabeled), add them into , then . For every , search the -nearest samples and their distances , then we can calculate the reconstruction by solving the optimization problem coefficients vector , if is smaller (11). Meanwhile, for every than the largest item in the th row of , replace the corresponding neighbor of with and update the reconstruction . After the label propagation matrix is coefficients vector updated, the new labels can be obtained by iterating (18) until convergence. It has been shown in [19] that many kernel algorithms are usually equivalent to conducting the linear algorithm in the kernel principal component analysis (KPCA) [20] transformed feature space. KLNP also has this property, that is, KLNP can be regarded as and achieved by KPCA followed by LNP. Next is an analysis on this issue. Suppose the th nonzero eigenvalue corresponding eigenvector calculated in the KPCA is , then the th projection of in the KPCA is . So the projections of to all computed KPCA projection directions can be represented as . After the projection, the reconstruction error in LNP can be represented in the KPCA transformed feature space as(19) 1: Using the RBF as the kernel, compute the kernel with respect to (when matrix , then , so is a sparse matrix as most samples are far away from each other); 2: Find the nearest neighbors of each sample in using the distance measure in (7), store the neighbors’ indices We can see that . Since the eigenvectors are orthogonal with each other, so . As the eigenvector are required to be normalized, i.e.,TANG et al.: VIDEO ANNOTATION BASED ON KLNP625TABLE III COMPARISONS OF RESULTS BY USING THE FEATURE EDGE DISTRIBUTION LAYOUTTABLE IV COMPARISONS OF THE MAPS BY USING THE FEATURE EDGE DISTRIBUTION LAYOUT[20], so obtain. We can (20)between (20) and (9), There is only a difference of constant so they are the same as a minimization problem. Now we can see that KLNP is equal to KPCA plus LNP. V. EXPERIMENTS To evaluate the proposed KLNP for video annotation, we conduct the experiments on the benchmark video corpus of the TRECVID 2005, which is consisted of about 170 hoursof TV news videos from 13 different programs in English, Arabic and Chinese [1]. We use the development (DEV) set of TRECVID05 in our experiments. After automatic shot boundary detection, the DEV set contains 43 907 shots. Some shots are further segmented into subshots, and there are 61 901 subshots for DEV set. These data are divided into two parts with 81% (50 000 subshots) as training set and 19% (11 901 subshots) as test set. For each subshot, 39 concepts are labeled as positive (“1”) or negative (“0”) according to LSCOM-Lite annotations [21]. These annotated concepts consist of a wide range of genres, including program category, setting/scene/site, people, object, activity, event, and graphics. Fig. 2 shows the exemplary thumbnails of these concepts. We adopt -NN here to find the neighboring points ( is set to 30 empirically). The process of searching neighbors and calculating coefficients is relatively time-consuming. It needs about 15 hours (with Intel P4 3.0G and 2G memory). However, this process is also required in other graph-based semi-supervised learning methods such as LNP and GRF. And fortunately it can be calculated offline. The parameters in these methods are all tuned to be nearly optimal through five-fold cross validations while is empirically set to 0.99 in consistency method. Once the label propagation coefficients are obtained, time costs of KLNP, GRF method and consistency method are all about one minutes, and LNP costs about 20 s. Although KLNP is a little slower than LNP, it is computationally effective as the processing time is always a challenging problem in TRECVID tasks. For each concept, systems are required to return ranked-lists of up to 2000 subshots, and system performance is measured via the official performance metric Average Precision (AP) [22] in the TRECVID tasks. The AP corresponds to the area under a noninterpolated recall/precision curve and it favors highly ranked relevant subshots. We average the APs over all the 39 concepts to create the Mean Average Precision (MAP), which is the overall evaluation result. The low level features we used here are: • 225-D block-wise color moments in LAB color space, which are extracted over 5 5 fixed grid partitions, each block is described by a 9-D feature; • 75-D edge distribution layout; • 144-D color correlogram in HSV color space. To avoid the curse of dimensionality, we conduct the experiments on the three feature sets respectively and then the three result scores of each sample for a certain concept are combined as a fused score using linear fusion through cross validations. For performance evaluation, we compare our algorithm with LNP [13] and other two popular semi-supervised learning626IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 10, NO. 4, JUNE 2008TABLE V COMPARISONS OF RESULTS BY USING THE FEATURE COLOR CORRELOGRAMTABLE VII COMPARISONS OF THE FUSED RESULTSTABLE VI COMPARISONS OF THE MAPS BY USING THE FEATURE COLOR CORRELOGRAMmethods: Gaussian random field [17] and consistency method [9]. The experimental results are shown in Table I (by using the feature color moments), Table III (by using the feature edge distribution layout), Table V (by using the feature color correlogram), and Table VII (for fused results) respectively.Comparing these results, we can see that KLNP performs the best on 27 of the all 39 concepts using the color moment, performs the best on 26 of the all 39 concepts using the edge distribution layout and performs the best on 23 of the all 39 concepts using the color correlogram. After linear fusion, KLNP performs the best on 27 of the all 39 concepts. The comparisons of MAPs between the KLNP, LNP, GRF, and the consistency method are shown in Table II, IV, VI and VIII, respectively. The MAP of KLNP is 0.27355 by using the feature color moment, which has an improvement of 3.14%, 8.86%, and 50.52% over LNP, GRF, and the consistency method, respectively. It has a MAP of 0.20604 by using the feature edge distribution layout, which respectively has an improvement of 5.51%, 8.35%, and 135.42% over LNP, GRFTANG et al.: VIDEO ANNOTATION BASED ON KLNP627TABLE VIII COMPARISONS OF THE MAPS OF FUSED RESULTSand consistency method. And it obtain a MAP of 0.24859 by using the feature color correlogram, which has an improvement of 6.17%, 4.97%, and 60.25% over LNP, GRF, and the consistency method, respectively. Through linear fusion, KLNP obtains a MAP of 0.30397, which has an improvement of 4.85% over LNP, 5.80% over GRF and 60.06% over the consistency method. All these comparisons demonstrate that KLNP is more appropriate than other semi-supervised learning methods and is effective for semantic video annotation. One thing should be mentioned, the performance of consistency method here is not as good as in many other applications [9]. The main reason is that we require the propagation matrix to be sparse for the large scale application and this requirement destroys the matrix’s symmetry which is needed in consistency method. VI. CONCLUSION We have analyzed the linear limitation of local semantics for LNP on ranking data with complex distribution, and proposed an improved method named KLNP, in which a nonlinear kernel-mapped space is introduced for optimizing coefficients reconstruction. The experiments conducted on the TRECVID dataset demonstrate that the proposed method is more appropriate for the data with complex distribution and is effective for the semantic video annotation task. REFERENCES[1] Guidelines for the Trecvid 2005 Evaluation [Online]. Available: http:// wwwnlpir. /projects/tv2005/tv2005.html [2] O. Chapelle, A. Zien, and B. Scholkopf, Semi-Supervised Learning. Cambridge, MA: MIT Press, 2006. [3] Y. Song, X.-S. Hua, L. Dai, and M. Wang, “Semi-automatic video annotation based on active learning with multiple complementary predictors,” in ACM Int. Workshop on Multimedia Information Retrieval, Singapore, Nov. 2005, pp. 97–104. [4] R. Yan and M. Naphade, “Semi-supervised cross feature learning for semantic concept detection in videos,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, San Diego, CA, Jul. 2005, pp. 657–663. [5] J. Tang, X.-S. Hua, G.-J. Qi, M. Wang, T. Mei, and X. Wu, “Structuresensitive manifold ranking for video concept detection,” in Proc. ACM Multimedia, Augsburg, Germany, Sep. 2007, pp. 852–861. [6] X. Yuan, X.-S. Hua, M. Wang, and X. Wu, “Manifold-ranking based video concept detection on large database and feature pool,” in Proc. ACM Multimedia, Santa Barbara, CA, Oct. 2006, pp. 623–626. [7] C. Wang, F. Jing, L. Zhang, and H.-J. Zhang, “Image annotation refinement using random walk with restarts,” in Proc. ACM Multimedia, Santa Barbara, CA, Oct. 2006, pp. 647–650. [8] J. Tang, X.-S. Hua, T. Mei, G.-J. Qi, and X. Wu, “Video annotation based on temporally consistent Gaussian random field,” Electron. Lett., vol. 43, no. 8, 2007. [9] D. Zhou, O. Bousquet, T. N. Lal, J. Weston, and B. Schölkopf, , S. Thrun, L. Saul, and B. Schölkopf, Eds., “Learning with local and global consistency,” in Advances in Neural Information Processing Systems 16. Cambridge, MA: MIT Press, 2004, pp. 321–328.[10] S. T. Roweis and L. K. Saul, “Nonlinear dimensionality reduction by locally linear embedding,” Science, vol. 290, pp. 2323–2326, 2000. [11] M. Belkin and P. Niyogi, , T. G. Dietterich, S. Becker, and Z. Ghahramani, Eds., “Laplacian eigenmaps and spectral techniques for embedding and clustering,” in Advances in Neural Information Processing System 14. Cambridge, MA: MIT Press, 2002, pp. 585–591. [12] L. K. Saul and S. T. Roweis, “Think globally, fit locally: Unsupervised learning of low dimensional manifolds,” J. Mach. Learning Res., pp. 119–155, 2003. [13] F. Wang and C. Zhang, “Label propagation through linear neighborhoods,” in Proc. 23rd Int. Conf. Machine Learning, Jun. 2006, pp. 985–992. [14] F. Wang, J. Wang, C. Zhang, and H. C. Shen, “Semi-supervised classification using linear neighborhood propagation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, New York, Jun. 2006, pp. 160–167. [15] B. Schölkopf, , T. K. Leen, T. G. Dietterich, and V. Tresp, Eds., “The kernel trick for distances,” in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2001, pp. 301–307. [16] J. Tang, X.-S. Hua, Y. Song, G.-J. Qi, and X. Wu, “Kernel-based linear neighborhood propagation for semantic video annotation,” in Proc. 11th Pacific-Asia Conf. Knowledge Discovery and Data Mining, Nanjing, China, Jun. 2007, pp. 793–800. [17] X. Zhu, Z. Ghahramani, and J. Lafferty, “Semi-supervised learning using Gaussian fields and harmonic function,” in Proc. 20th Int. Conf. Machine Learning, Washington, DC, Aug. 2003, pp. 912–919. [18] D. Tao, X. Tang, X. Li, and Y. Rui, “Direct kernel biased discriminant analysis: A new content-based image retrieval relevance feedback algorithm,” IEEE Trans. Multimedia, vol. 8, no. 4, pp. 716–727, Aug. 2006. [19] D. Tao, X. Li, and S. Maybank, “Negative samples analysis in relevance feedback,” IEEE Trans. Knowl. Data Eng., vol. 19, no. 4, pp. 568–580, Apr. 2007. [20] B. Scholkopf, A. Smola, and K.-R. Muller, “Nonlinear component analysis as a kernel eigenvalue problem,” Neural Comput., vol. 10, no. 5, pp. 1299–1319, 1998. [21] M. Naphade, J. R. Smith, J. Tesic, S.-F. Chang, W. Hsu, L. Kennedy, A. Hauptmann, and J. Curtis, “Large-scale concept ontology for multimedia,” IEEE Multimedia Mag., vol. 16, no. 3, pp. 86–91, Jul.–Sep. 2006. [22] Trec-10 Proceedings Appendix on Common Evaluation Measures [Online]. Available: /pubs/trec10/appendices/measures.pdfJinhui Tang (S’04) received the B.S. degree in 2003 from the University of Science and Technology of China, Hefei, where he is currently pursuing the Ph.D. degree in the Department of Electronic Engineering and Information Science. From June 2006 to February 2007, he was a research intern with the Internet Media Group at Microsoft Research Asia. From February 2008 to May 2008, he was a research intern with the School of Computing, National University of Singapore. His current research interests include content-based video analysis, pattern recognition, and image retrieval. Mr. Tang is a student member of the Association for Computing Machinery.Xian-Sheng Hua (M’05) received the B.S. and Ph.D. degrees from Beijing University, Beijing, China, in 1996 and 2001, respectively, both in applied mathematics. Since 2001, he has been with Microsoft Research Asia, where he is currently a Lead Researcher in the Internet Media Group. His research interests are in the areas of video search, content-based video analysis, pattern recognition, and machine learning. He has authored more than 100 publications in these areas and has 20 patents or pending applications. Dr. Hua is an Associate Editor of IEEE TRANSACTIONS ON MULTIMEDIA and has been on the Editorial Board of the Journal of Multimedia Tools and Application since 2007. He is a member of the Association for Computing Machinery.。