[2010]Cross-media Retrieval Based on CSRN Clustering

Journal of Computational Information Systems 6:9(2010) 2821-2830Available at Cross-media Retrieval Based on CSRN ClusteringCheng ZENG1,†, Zhenzhen WANG1, Gang DU21State Key Lab of Software Engineering, Wuhan University, Wuhan, China, 4300722School of Computer, Wuhan University, Wuhan, China, 430072AbstractA novel cross-media retrieval methodology is proposed to help user more accurately and naturally describe theirrequirement and gain response. The first problem to be solved in this paper is how to build the bridge among heterogeneous information. An efficient approach is adopted to mine the semantic relationship among different multimedia data. We propose the scheme, called cross-media semantic relationship network (CSRN), to store the cross-media relationship knowledge, which is constructed by mining various potential relation information in the Internet. By hierarchical fuzzy clustering based on semantic relationship against CSRN, we obtain semantic vector bundle which could gather different modalities but same semantic of diverse media feature vectors. Furthermore, dynamic linear hash index is used in these vectors to adapt to the flexible expanding or updating of CSRN and quickly retrieval based on multiple examples without the limit of media modality by hash intersection. We show through experiment based on different modalities or different numbers of examples that our approach can provide more flexible retrieval mode and more accurate retrieval results.Keywords:Cross Media; Semantic Relationship; Semantic Vector Bundle; Hierarchical Fuzzy Clustering1. IntroductionCommercial multimedia search engines consider more on efficiency and realize search based on relevant technique of keywords or low-level features matching, such as Google.image, Bing.image and Like, GazoPa, etc. However, these systems provide only simple requirement expressing way to user so that user’s intention is difficult to be understood and the results are always poor amd single modality. On the contrary, academic circles are exploring better man-machine information exchanging technology, where the machine would adapt to the human’s habit of expressing and receiving information as much as possible, but not that the human have to adapt to the understanding ability of machine. Lots of new research topics emerge which generally refer to the concept “semantic”, such as semantic search, semantic multimedia, semantic computing and so on. Their common ground is to attempt to create, process or display information with the thinking way as human, e.g. IGroup[1] provides the semantic classifications of results. Hakia, Cuil, Kosmix and PlayAudioVideo support multimodal of results displaying, and Powerset focuses on natural language search. In addition, there are so many techniques[2-5] which realize semantic or personalized retrieval based on ontology.Many earlier cross-media studies [6,7] focused on integrating or managing heterogeneous multimedia data. Henceforth, cross-media technology is applied into diverse multimedia retrieval domain. Liu [8] proposes a cross-media manifold co-training mechanism between keyword-based metric space and vision-based metric space and deduce a best dual-space fusion by minimizing manifold-based visual & textual energy criterion. At last, he realizes image annotation and retrieval. In [9], a dual cross-media †Corresponding author.Email addresses: zengc@ (Cheng ZENG), zhenzhen_1987cn@ (Zhenzhen WANG),melephas@ (Gang DU)1553-9105/ Copyright © 2010 Binary Information PressSeptember, 20102822 C. Zeng et al. /Journal of Computational Information Systems 6:9(2010) 2821-2830 relevance model which considers both word-to-image and word-to-word relations is proposed to automatically annotate and retrieval images. Furthermore, there are lots of researches on video retrieval based on cross-media. Bruno et al. [10] designs a multimodal dissimilar space which is constructed based on positive and negative examples distribution and realizes video retrieval by space clustering and modalities fusing. Meanwhile, many researchers explore cross-media comprehensive retrieval way, such as a typical application about E-learning [11]. [12] utilizes the complementarity among different modalities of media data to mine co-occurrence relationships of news items, and then provides cross-media content complementation. Zhuang et al. [13,14] integrate the webpage structure analyzing and low-level features correlation mining of multimedia data to construct cross-media relationship graph UCCG, and short-term and long-term feedbacks play vital role in the method.Cross-media technology aims at making maximum use of relevance, cooperation, complementarity among diverse modalities of media to fuse all sorts of information. As a result, user can gain the desired information with higher accuracy, more quickly speed and lower cost. The kernel of realizing cross-media retrieval is to discover the semantic relationship among different multimedia data. However, the semantic “gap” and the feature diversity among various modalities of media make the development of this domain difficult in taking a step. Furthermore, the common character of high information density of multimedia data lead to that the relevant processes generally require a large amount of computing time and resources. It is very difficult to develop a universal cross-media retrieval engine at present. However, the technology has good application prospect in some certain domain, such as E-learning, E-business, tourism and entertainment, etc.2. Framework of Cross-media Retrieval SystemThis paper gains semantic relationships among diverse data from internet where multitudinous knowledge is contributed by various persons or software systems every moment. Initially, we don’t concern about the modality of data, but pay attention on the channels where we can get the relationship. All media objects, which are the results by processing multimedia document and semantic relationship among them will be stored into a unified model, called cross-media semantic relationship network (CSRN). For multimodal, time-based and text types of multimedia document, we segment them into the smaller granularity of unit. The kernel of our approach is how to construct and utilize CSRN. We give the definition of CSRN below: Definition 1. Let each node represent a modality of media object, the metadata structure of is as follows:, , , , , , , , , (1) Each edge , denotes semantic relationship between different media objects, which comes from 4 aspects:processes space dependency of web data based on visual structure and hyperlink in the webpage, e.g. adjacent text, image or other modality of media objects in a webpage have more similar semantic;analyzes the labels and ranking of search results coming from multiple multimedia search engines based on the same search condition;mines the inherent relevancy among different modalities of data or same modality of segmented data in a multimedia document, e.g. visual, audio and textual data in a news video, or all video scene clips in a sport video have certain relevancy each other;accumulates potential user feedback during the process of user browsing. E.g. those results clicked one after another generally have approximative topic or similar semantic.We call the the integration of these nodes and edges as CSRN.In formula (1), each media object has a globally unique identity SID and GID corresponds to the document ID. Moreover, not each media object corresponds to an example. Time-based media such as video, audio possibly contain too much semantic in a same document, which possibly lead to most media objects having semantic relaionship each other. Overly abundant semantic relationship in CSRN is equivalent to not be able to gain any valuable information. Moreover, a multimedia document includingC. Zeng et al. /Journal of Computational Information Systems 6:9(2010) 2821-2830 2823multiple modalities of data will also affect the analysis. Thus, it’s necessary to previously split or segment these multimedia documents before they are regarded as media objects. Likewise, text is segmented based on paragraph to reduce the semantic complexity of each media object.After CSRN is constructed, the hierarchical fuzzy clustering will be used on CSRN. The purpose is to put all media objects with similar semantic into same cluster, called semantic category (SC). It is possible that a media object belongs to multiple SC, and even several media objects belong to distinct or same SC in different conditions. Thus, the hierarchical fuzzy clustering method will be suitable and essential to these statuses. All media objects will be substituted for their feature vectors. Because there are a large number of different modalities of media object in same SC and many of them possess similar low-level features which are actually redundant, we quantify all feature vectors in a SC by feature vectors clustering against different modalities of media objects, respectively. Then the typical feature vectors(TV), called semantic vector bundle (SVB), will be selected to represent the SC. It means that we obtain the mapping among all possible feature vectors of different modalities but similar semantic. At last, hash indexs will be respectively created on these TV sets corresponding to different modalities of media objects. Thus, if a user submits several media examples with same or different modalities, they will be preprocessed and transformed into feature vectors, and then respectively locate their own SC sets. The architecture of our cross-media retrieval method is shown in Fig.1.3. Cross-media Semantic Relationship NetworkA great deal of researchers improve the system performance by mining implicit or explicit knowledge in the internet. Furthermore, most of commercial search engines use keywords and hyperlink extracted from Webpages to index the images, videos or other data. In our approach, all initial cross-media semanticrelationship knowledge comes from internet and these knowledge will be stored in CSRN.Fig.1 Architecture of Cross-media Retrieval System2824 C. Zeng et al. /Journal of Computational Information Systems 6:9(2010) 2821-28303.1. Webpage Visual Structures and Hyperlink AnalysisSome privious methods regard webpages as information units. A webpage generally contains multiple and different granularities of semantic so that these methods will confuse the semantic correlation among media objects. VIPS algorithm [15] is proposed to segment webpage by analyzing DOM tree based on visual structure. The highest sensitive granularity of VIPS is <TD> level because its purpose is to segment webpage block. But in our work, extracting the media objects and the relationship among them are the final purpose. Hence we improve VIPS algorithm so that it is sensitive to URL address, <BR> in continuous text and hyperlink etc.. Thus, each leaf node will actually correspond to a media object and the semantic relationship only exist between any two leaf nodes. Fig.2 shows an example of webpage visual segmentation and the final visual structure tree constructed by improved VIPS is shown in Fig.3.Fig.2 An Example of Webpage Visual Segmentation Fig.3 An Example of Webpage Visual Structure TreeThe similarity between any two nodes can be calculated based on the shortest path in the tree. Becausesemantic generalization will lose some semantic information while semantic specialization narrows therange of semantic description, we give different semantic attenuation constants α and βrespectivelycorresponding to upward and downward section in the path. The formula of relationship calculatingbetween any two media object and is as follows:, 1 0 1 0 (2) where n and m denote the numbers of upward and downward sections, respectively.3.2. Results Analysis of Multimedia Search EnginesMultimedia search engines return lots of related multimedia documents based on keywords, such asgoogle.image, Youtube, Flickr etc.. Hakia, Cuil even simultaneously returns different types of multimedia documents. A concept set is extracted from LSCOM, and then the set is extended with WordNet. These concepts will be selected or combined according to some certain rules in turn and be simultaneouslysubmited to all search engines. For each search, all returned results are expected to have similar semantic, and a sub-CSRN could be constructed. But polysemy confuses the results and affects the ranking.Contextual concepts in result labels are used to filter most confusing results. Thus, we syntheticallyconsider the result label which are preprocessed to only reserve noun and verb keywords, and result rankingof search engine to measure the semantic relationship among returned results.At first, , which denotes semantic similarity between search keywords set and concepts set corresponding to v th result label will be calculated:, , , ,,, , (3)In the above formula, KM denotes classical Kuhn-Munkres algorithm, and , denotes the semantic similarity between any two concepts , in and , respectively, and 0 , 0 where , are represented as the amount of retained search keywords and correspondingreturned results, respectively., represents the number of same concept between and , while,is the amount of all different concepts in and . is constant between 0 and 1 when theC. Zeng et al. /Journal of Computational Information Systems 6:9(2010) 2821-2830 2825 intersection between and is null after deleting same concepts from them. Semantic similarity function [16] based on WordNet is utilized to calculate the similarity between different concepts.,1 2 , ,, / , , ,(4) Detailed mathematical introduction of formula (4) can be found in [16]. We previously construct a concept similarity map (CSM) for avoiding repeatedly computing , .We define semantic relationship, of any two results (media objects) with same search keywords in formula (5). It is measured based on the cosine of angles between two result vectors constructed in the results semantic similarity space (R3S) where horizontal ordinate denotes the similarity degree between result label and search keywords, and vertical ordinate denotes ranking value., , ,, ,0 , (5)where , are decimal value which denote the results of ranking number i and j divided by , respectively.3.3. Other Aspects of Semantic Relationship and Their SynthesisFor mining aspect of relationship, we require to adjust the size of each time-based or multimodal media object in initial CSRN or even decompose it based on modality so as to simplify its semantic. Currently, there are lots of mature technique to support splitting diverse modalities of data from multimodal document, such as splitting audio from video. Then different modalities of time-based data contained in the same document will be jointly segmented with same boundary, where audio data are segmented referring to the results of video segmentation based on scene. Each video or audio segment would eventually correspond to a node (media object) in CSRN. The semantic relationship between different modalities of data segment within same time range are always 1, namely, 1. While semantic relationship between different data segments with same modality is defined based on adjacency distance:,1 01√ 1⁄ 1 (6)where x denotes the interval number of segment between and , and is a constant representing degeneration degree of semantic relationship.aspect will be able to optimize the relationship knowledge in CSRN after each time of cross-media search feedback, such as browsing, printing, downloading, etc., where the system will save operating results with same search condition. These are regarded as implicit user contributing. The relationship among the results will be used to update CSRN:, ∑ · 2, 0 (7)where denotes the amount that and have same operation in same search condition and represents the influence degree of each kind of operation. If the result of updated, is greater than 1, it will be directly replaced by 1.4. CSRN Index for Efficiently Retrieval and ExpandingYou could submit an image example and find a most similar image in CSRN with traditional feature matching technique, and then gain related diverse modalities of multimedia documents based on cross-media semantic relationship. However, when user submits several examples which are heterogeneous/homogeneous, the above way is possible to return several irrelative results sets. Furthermore, the retrieval efficiency will be low and CSRN expanding is difficult when new multimedia documents are inserted. Thus, we will introduce an efficient cross-media index mechanism in this section.2826 C. Zeng et al. /Journal of Computational Information Systems 6:9(2010) 2821-28304.1. Hierarchical Fuzzy ClusteringBy clustering on CSRN, all media objects with similar semantic can be congregated into the same semantic category (SC). However, most media objects possess complex semantic, and they are impossible to simply belong to a certain SC. Moreover, the clustering granularity is difficult to make certain. If the granularity is fine, we can obtain relatively accurate results but lose the recall ratio. Thus, it’s necessary to use hierarchical fuzzy clustering (HFC) algorithm against CSRN.The whole process of HFC includes two phases. In the first phase, K-distance neighbor is used to construct the initial status of HFC. Each media object and its K-distanced neighbors set can be picked out easily from CSRN which is stored with symmetric matrix. The average distance between and any object in can be calculated and denoted as . Then the relative density of K-distance neighbor of object is defined as follows:,…, , ,…,,…, , ,…, (8)Given the threshold value 0, is considered as a core object when 1 . For with , core objects set is defined as | where O is the list of entire core objects } which is the initial status of any cluster. In other words, a core object with current global minimum and without class mark will be picked out, and is obtained to form a new marked cluster . Then all objects in of cluster will be gradually extended by adding the K-distance neighbors of each core object in turn. Assume that is one core object in cluster , which has not been extended, and its K-distance neighbors can be obtained, denoted as 1,…, . If is a core object of other cluster, it will be ignored. Otherwise, it will be added into cluster . The cluster can be extended until no more objects in it can be extended. The process continues until all core objects have been extended. Here we can find that core objects are pivotal for each cluster and it is impossible that a core object belongs to more than one cluster, while other objects can exist in multiple clusters in the same time. During the course of extension, two clusters will be marked as adjacent clusters when part of objects they include are the same. Thus, we obtain the highest granularity of fuzzy clustering result.In the second phase, the clusters obtained in the first phase will be fuzzily clustered further. First, fuzzy similarity , of any two adjacent clusters and is calculated as:, ,, (9)Then descending threshold value set 1,…, is selected for measuring similariy from fine granularity to coarse granularity. For each cluster , similar cluster set under certain granularity level ∆ is defined as | , . The clusters in can be united into one cluster on level ∆. As the value of descend, we can obtain hierarchical clustering of objects and the granularities of clustering are from fine to coarse, until they are united into one cluster.4.2. Index on CSRNMedia feature vector quantification is the first process step which does not reduce the dimension number of vector but the amount of vectors. However, the dimension and implication of heterogeneous media feature vectors are different. Thus, we firstly distinguish feature vectors according to media modality, and then cluster these vectors with FEC algorithm [17], and the average vector (called typical vector, TV) is regarded as the representative of a cluster, shown in Fig.4. All TVs in same SC make up a semantic vector bundle (SVB). Each SVB enumerates various feature vectors which come from different modalities of media objects but represent similar semantic. We build the mapping relation , between TV and feature vectors in corrrespondng cluster and , between TV and SC where , σ respectively denotes the granularity level in HFC and media modality.For flexibly expanding or updating CSRN, and efficiently searching feature vector, we create dynamic linear hash index on which computes the hash value of TV only one times, stores hash values usingC. Zeng et al. /Journal of Computational Information Systems 6:9(2010) 2821-2830 2827as few buckets as possible and linearly increases buckets. Several vector hash functions are respectively defined for different dimension of vector corresponding to different media modality . The actual active bit number of could be calculated as follows, which always use the low order bits:(10)Where is represented as the amount of TV of modality σ.If a user submits an example , the system will extract its feature vector at first, and then invoke the corresponding . The returned result determines a unique hash bucket which stores a TV set having same hash value. The media object whose feature vector is most similar with must also exist in this bucket. Then, we use vector matching technique to locate the media object. Suppose Top( ) denoted the most similar TV with in hash bucket . We can get many SC sets of different granularity levels based on , .∑, (11) By now, user actually gains a result set containing multiple modalities of media objects which have been classified through SC in different granularity levels. is the highest granularity. If a user pays attention on precision, he can select fine granularity. When user submits multiple examples which are even different modalities, we use hash intersection to realize search. The search examples set is represented as . The system returns a corresponding to each example so that we can calculate the intersection among these SC sets of different examples in each granularity level.∑, (12)5. Experiments5.1. Data sources and CSRNOur experiments data all come from internet through webpage analysis which includes two aspects of pages. The first is the common pages of Web encyclopedia 1, Yahoo!news, etc.. We manually select the initial pages as entrance, and then the related webpages will be automatically crawled based on hyperlinks. The another is results page of multimedia search engines such as Google.image, Hakia, Flickr, etc. We extract 118 concepts with distinct visual or audio characteristic from LSCOM and expand them with synonym and cognate words in WordNet. At last, 203 concepts are used and they compose into 1065 keywords set, each set of which includes 1 to 3 concepts. These keywords set are simultaneously submited to all search engines in turn and are projected into the same R3S to calculate results similarity each other for same search condition. For ensuring relatively high accuracy and avoiding overfull results set, we only analyze and store 1; ; .Fig.4 The Schematic Diagram of CSRN Index2828 the top from 3some a minute For (mean,contras and th feature 5.2. Th Fig.5(1multip text by as lowest quantit contain semant video a and ge ways t higher filters reduce examp For by mu previou multip examp modali audio m precisi consid of com search 2http://w C p 15 results fo 352 video, 134audio material e data for all ti image, we ex , variance, ske st, Orientation he motion fea es slope, roll-o he Performan 1)-(3) show th le same moda y image as ex spect where te precision due ty in asp ns the feature tic relationshi and audio hav enerate relation to get audio d average searc the semantic e the deviatio les is similar a (1) Cross-media Image ExaFig.better reflecti ultiple modalit us dataset, sh le modalities les and more ities of media media data as ion of image+erable role du mbinational ex examples as/sm C. Zeng et al. /or each search 4,798 text par l websites suc ime-based me xtract 3 types o ewness), and 3n). For video ature histogra off, centroid an nce of Cross-m he experimen ality of media xample has the ext and imag e to less relati ect. Although e in key fram ip informatio ve better searc nship in data. It can be ch accuracies distinction am n between se as Zhuang’s a Retrieval by ample 5 Comparison of ng the charac ties of media hown in Fig.6s of example e modalities o a data as exam s examples ha +audio and v uring the proce xamples. Thes many as posmzy/dongwu-yx-7/Journal of Co h. We totally c ragraphs and 2ch as smzy 2 an edia.of color featur 3 types of text segment, we am in whole nd OER are u media Retrieva nt results of q a data as exam e highest prec e are in the m ionship which h video has sam me which is th n about imag h accuracies e aspect. But th e seen that m than single d mong differen earch example approach [14], b (2) f Cross-media Ret teristic of ind a data combin 6. We can ob es generally r of examples c mples, we can ave higher ac video+audio ess of cross-m se phenomeno sible will mak711-6.htmlomputational Icollect 40,610 2,809 audios nd so on. We re including c ture features i mainly reserv segment. For used.alquerying diffe mples which a cision because majority. But h are mainly l me situation a he same as an ge and enhan each other bec his situation w multiple docum document in a nt examples an es and user r ut we do not r Cross-media Ret Video Example trieval among Di ex intersection nation as exam bserve severa receives high can receive hi still get good ccuracy than p are higher th media retrieval on demonstratke the system Information S images and 3which are ext filter voice in color histogram including MR ve the feature r audio, cepst erent modaliti are all out of e we extract a querying aud limited in as audio, the fe n image. So a nce relational cause most of will be possibl ments as exam ll three figure nd receives th requirement. rely on iterativ trieval by e fferent Modalitie n in our appro mples where al interesting her search ac igher accuracy d results. Seco pure visual m han that of im l which signif te the principlm more accura Systems 6:9(203,379 video se tracted from v n audio and on m, color cohe RSAR, Gabor, e of key fram tral features ies of media our dataset. I a mass of sem dio by image and asp feature vector actually, video search accur f audio data ar ly changed if mples together es owing to SC heir semantic The search a ve short-term (3) Cross- Au es of Search Exam oach, we test c all examples phenomenon ccuracy than y. Even thoug ond, the comb media combina mage+video. ficantly raises le that using vately understa 010) 2821-283egments which videos or com nly analyze the rence, color m Tamura (coar me regarded as MFCC and s objects by sin In Fig.5(1), qu mantic relation as example h pects and has extracted from o indirectly u racy. In Fig.5e splitted from f we have mor r receive rema C intersection in common s accuracy of m feedback.-media Retrieval udio Example mples cross-media re are also out n. First, query single moda gh we only u ination of visu ation, e.g. the Third, text p s the search pr various modaland user requi 30h come me frome first 1moment rseness, s image spectral ngle or uerying nship in has the a small m video uses the (2)-(3), m video re other arkably n which so as to multiplebyetrieval t of the ying by ality of use two ual and search plays a recision lities ofirement。