First Steps to an Audio Ontology-Based Classifier for

First Steps to an Audio Ontology-Based Classifier forTelemedicineCong Phuong Nguyen, Ngoc Yen Pham, Eric CastelliInternational Research Center MICAHUT – CNRS/UMI2954 – INPGrenoble1, Dai Co Viet, Hanoi, Vietnam{Cong-Phuong.Nguyen, Ngoc-Yen.Pham,Eric.Castelli}@.vnAbstract. Our work is within the framework of studying and implementing asound analysis system in a telemedicine project. The task of this system is todetect situations of distress in a patient’s room based on sound analysis. If sucha situation is detected, an alarm will be automatically sent to the medical cen-tre. In this paper we present our works on building domain ontologies of suchsituations. They gather abstract concepts of sounds and these concepts, alongwith their properties and instances, are represented by a neural network. Theontology-based classifer uses outputs of networks to identify classes of audioscenes. The system is tested with a database extracted from films.1 IntroductionIn recent years telemedicine is widely studied and applied. It can be broadly defined as the transfer (e.g. telephone lines, the Internet, satellites, etc) of electronic medical data (e.g. images, sounds, live video, patient records, etc) from one location to an-other. The system that we present is developed for the surveillance of elderly, conva-lescent persons or pregnant women. Its main goal is to detect serious accidents such as falls or faintness at any place in the apartment. If a serious accident is detected, an alarm will be automatically sent to the medical centre. Firstly most people do not like to be supervised by cameras all day long while the presence of microphone can be acceptable. Secondly the supervision field of a microphone is larger than that of a camera. Thirdly, sound processing is much less time consuming than image process-ing, hence a real time processing solution can be easier to develop. Thus, the original-ity of our approach consists in replacing the video camera by a system of multichan-nel sound acquisition. The system analyzes in real time the sound environment of the apartment and detects abnormal sounds (falls of objects or patient, scream, groan) that could indicate a distress situation in the habitat.Fig. 1. The apartment used for telemedicine. Microphones are installed in each room in order to assure the sound surveillance in the whole apartmentThis system is divided into different small modules. In the process of developing it, a sound acquisition module, a sound of everyday’s life classifier, a speech/nonspeech discriminator and a speech/scream-groan discriminator are con-structed [10], [11], [12], [18]. The habitat we use for our experiments is a 30m2 apartment (depicted in Fig. 1) equiped with various sensors, especially microphones. There is one microphone in each room (toilet, kitchen, shower-room, hall and living-room) of the apartment. This installation allows a sound surveillance in the whole apartment. In this apartment, audio signals are acquired by five microphones and feed to a multichannel data acquisition card (National Instruments DAQ) installed on a slave computer. Sound source can be localized through comparison of the soun levels of the microphones. If two simultaneous detections are recorded, only the channel with the maximum signal level is considered. The microphones used are omni-directional, condenser type, small size and low cost. A signal conditional card con-sisting of an amplifier and an anti-aliasing filter is attached to each microphone. The four modules mentioned above, developed in LabWindows/CVI, process acquiredsignals to detect situations of distress.The combination of these four modules seems to be an audio classification system. It can be seen that audio classification has been studied for many years. They are applied to speech recognition, audio content-based analysis, audio segmentation, audio retrieval, broadcast news transcription, etc. Works described in [22], [4], [16], [28] are four examples among many systems. But there is a difference between the problem of audio classification and our problem. Audio classification in the literature is applied to classify different homogeneous audio segments, i.e. each segment con-sists of a unique type of audio signal (e.g. speech). Meanwhile, our system is intended to classify an audio scene containing different types (e.g. a segment of scream and a segment of fallen chair). In other words, an audio scene’s category is determined by its types of segment. In order to complete the system, we propose sound ontologies which can be used in an ontology-based classifier. Each ontology is an abstract con-cept of an audio scene representing a situation of distress in the house. It can be used to detect situations of distress, to classify audio scenes, to share informations of audio scenes among people and software, to analyze domain knowledge, or to save (as metadata) audio scenes in a database for further usages.This article is structured as follows. Sect. 2 discusses works related to ontology-based audio applications. Sect. 3 describes the proposed ontology of audio scenes and the neural network used to represent ontologies and the ontology-based classifier. Sect. 4 presents the database of audio scene and the evaluation of ontologies. Sect. 5 outlines our conclusion and next steps in future to complete this ontology-based sys-tem.2 Related WorkOntology has been researched and developed for years. In audio applications, it is applied probably for the first time by Nakatani and Okuno [19]. They propose a sound ontology and its three usages: ontology-based integration (for sound stream segregation), interfacing between speech processing systems, and integration of bot-tom-up and top-down processing. Their sound stream segregation means generating an instance of a sound class and extracting its attributes from an input sound mixture. Khan and McLeod [13] utilize a domain-specific ontology for the generation of meta-data for audio and the selection of audio information in a query system. In this work, an audio ontology is defined by its identifier, start time, end time, description (a set of tags or labels) and the audio data. MPEG-7 Description Definition Language and a taxonomy of sound categories are employed by Casey [5] for sound recognitions. The audio content is described by qualitative descriptors (taxonomy of sound categories) and quantitative descriptors (set of features). Amatriain and Herrera [1] use semantic descriptors for sound ontology to transmit audio contents. Their description includes both low-level descriptors (e.g. fundamental frequency) and high-level descriptors (e.g. 'loud’). WordNet, an existing lexical network, is used by Cano et al. in [3] as a ontology-backbone of a sound effects management system. Ontology is applied to the disambiguation of the terms used to label a database in order to define concepts of sound effects, for example, the sound of a jaguar and the sound of a Jaguar car. A system of ontology-based sound retrieval is proposed by Hatala et al. in [9] to servemuseum visitors. Ontology is used to describe concepts and characteristics of sound objects as an interface between users and audio database. In [14], Kim et al. develop an MPEG-7-based audio classification and retrieval system targeted for analysis of film material. They test three structures for sound classification: an one-level, a hier-archical, and a hierarchical with hints. The classification is based on the Euclidean distance. The second structure gives the lowest recognition rate, while the third one gives the highest recognition rate. In most cases, ontology is used to describe an au-dio file (e.g. a sound effect) and to manage the database.Our work is to build a sound ontology applied to classifying an unknown audio sample detected in habitat. We will present in the next section ontologies for abstract concepts of audio scenes and for detecting situations of distress.3 Ontology-Based Sound ClassificationAn ontology-based sound classification includes three problems: defining ontologies, representing them, and applying them to classification. These problems will be pre-sented in this section.3.1 Sound OntologyFrom the classified sounds, we intend to extract the true meaning of the scene. For example, when a sound of a fallen chair and a sound of scream are detected, it should be interpreted as “the patient has tumbled down” (making the chair fall). Or when we detect a sound of groan, we can say that the patient is ill or hurt. This is a mapping between concrete sounds and abstract concepts. In other words, an abstract concept is defined by determined sounds. Sound ontology seems to be the most appropriate to our work.An ontology is a definition of a set of representational terms as defined by Gruber in [8]. In this paper, some situations are defined as ontologies. An ontology consists of concept, concept properties, relations and instances. The hierarchical relations between concepts in text or image applications can be established. But in our applica-tions, such relations between situation are not obvious. So we simply define concepts of situations based on concept properties and their facets. The ontology of the situa-tion of patient falling in house is depicted in Fig. 2 as an example. When the patient falls, he can make a chair or a table fall over, or can break something such as a glass. After the fall, the patient probably screams or groans due to pain or shock. Sounds are divided into two categories. Sounds of fallen chair, fallen table and broken glass are categorized in the “thing_distress” class, sounds of scream and groan are of “hu-man_distress”. So if a sound of “thing_distress” and/or a sound of “human_distress” are successively detected, we can probably say that the patient has fallen. Of course it can also be said that those sounds come from a chair tumbled down by a man and from another man being hurt. But if we suppose that normally the patient lives alone in the house then the “fall” interpretation is the most appropriate. Two other concepts of hurt and sick/ill are listed in Table 1. It is noted that water_in_toilet includes soundof water discharge, of water flowing from shower, water rushing or dripping fromfaucet, etc. In short, the concept of sound can be identified based on its attached con-cept properties and their respective facets. The usage of these concepts will be pre-sented in next paragraph.Fig. 2. An example of the ontology representing the “fall” concept. It has two properties andfive facetsTable 1. Properties and facets of three ontologies. “Hurt” concept has no thing_distress prop-erty. “Water_in_toilet” facet taken into account is due to the fact that when the patient issick/ill in the toilet, sound of water is often detectedThing_distress Human_distressFall Fallen_chair, Fallen_table, Broken_glassScream,GroanHurt Scream,GroanSick/ill Water_in_toilet Cough,Pant,Vomit3.2 Ontology-Based Sound ClassificationOntology-based classifications are mostly applied in text, image and biological appli-cations. The image classification system presented by Breen et al. in [2] uses a neural network to identify objects in a sports image. Category of the image is determined by a combination of detected image objects, each objects being assigned an experimental weight. Image ontology in this case is used to prune the selection of concepts: if the parent and the children are selected, the later will be discarded. In order to automati-cally classify web page, Prabowo et al in [23] apply a feed-forward neural network represent relationships in an ontology. The output of this network is used to estimate similarity between web pages. Mezaris et al. in [18] propose an object ontology for an object-based image retrieval system. In this ontology, each immediate-level descrip-tors is mapped to an appropriate range of values of the corresponding low-level arith-metic feature. Based on low-level features and query keywords, a support vector machine will result the final query output. In [21], Noh et al. classify web pages using an ontology. In this ontology, each class is predefined by a certain keyword and their relations. Classes of web pages are classified by extracting term frequency, document frequency and information gain, and by using several machine learning algorithms. Taghva et al. in [25] construct an email classification system in which an ontology is applied to extract useful feature, these feature are inputs of a Bayesian classifier. The text categorization system of Wu et al. in [27] also employ an ontology predefined by keywords and semantic relationship of word pair. These keywords are chosen by a term frequency / inverse document frequency classifier. The domain of a text is cate-gorized by a “score”. Maillot et al. in [17] introduce their ontology-based application for image retrieval. Each image object in an image is detected by a trained detectors. The relations of detected image object, established in ontology, will determine the category. In [26], Wallace et al. present their multimedia content indexing and re-trieval system. The ontology (basing on MPEG-7 description schemes) of the system includes semantic entities, semantic relations and a theraurus. A text-mining-based ontology enhancement and query-processing system is presented by Dey and Abu-laish [6]. Their key ideas are to learn and to include imprecise concept descriptions into ontology structures. Instead of using “very sweet” and “intensely sweet” in the wine ontology, they assign “sweet” for both wines, “very” for the first and “in-tensely” for the second. The degree of similarity between “very” and “intensely” is computed by a fuzzy reasoner. In [15], Laegreid et al. present a system for biological process classifications using Gene Ontology (GO, stored at Gene Ontology Home page, [7]). Their goal is to model the relationships between gene expression and a biological process, to learn and classify multiple biological process roles, and to use this model to predict the biological participation of unknown genes. Their method uses biological knowledge expressed by GO and then generates the model. In work of Robinson et al. [24], results of cluster analysis of gene expression microarray data is based on GO terms and associations.In ontology-based image applications, an image object is often defined by lower properties, such as form, color, viewing angle, background, etc. The method of defin-ing related lower properties in image applications is hard to apply to audio domain, because an audio object, such as a laugh, is hard to be defined.Text applications use relationships between text objects to classify. They are often known or predefined. For example, in a text application “net” can be interpreted as “a fishing tool” if words such as “fish”, “boat”, “river” are found in the same sentence of paragraph because they have obvious relation; or it can be a “group of connected computers” if there are “port”, “cable”, “TCP/IP”, “wi-fi” nearby.GO is usually applied as a basis of biological applications. It provides a structured and controlled vocabulary describing genes and gene products in organisms. This ontology consists of biological objectives, molecular functions and cellular compo-nents, organized into hierachies. To our knowledge, audio applications can take no advantage of this type of ontology.In our work methods used in text applications are considered because we hope to find the meaning of an audio scene by a group of sounds. The predefined classifying rules of Noh et al. is hard to apply to audio domain because so far we do not know a predefined rule for sounds. The keyword-based ontology of Wu et al. is also difficult to be used in our work because it needs relationship between sounds. The imprecise concept descriptions of Dey and Abulaish demand adjectives and adverbs. It cannot be applied to audio domain since audio signal has no equivalence. The method of using neural netwok to represent ontology of Prabowo et al. seems to be the most appropriate for us since it demands only two levels of concept and properties. There-fore in our work the relationship among concept, concept properties and facets is represented by a feed forward neural network described in [23]. The task of this net-work is to produce an output value that is used to estimate the similarity between the ontology and a new sound situation.Instance Instance InstanceFig. 3. The feed-forward neural network representing the “fall” ontology. Weights of links between layers depend on number of properties, number of facets and number of detected instancesThis model has three layers: input, hidden and output. The input layer of the net-work represents a set of instances. The hidden layer represents a set of concept prop-erties. The number of neurons in the hidden layer equals the number of concept prop-erties. The output layer (consists of one neuron) is the concept representatives. The two reasons to choose sigmoid transfer function,f(x) = 1/(1 + e-x) (1) for neurons, are as follows. Different concepts have different numbers of proper-ties, so their outputs are normalised form 0 to 1. And they vary continously. The neural network representing the “fall” concept is depicted in Fig. 3. In this example there are two hidden neurons and five input neurons. Two properties “thing_distress” and “human_distress” are represented by two hidden neurons. Input neurons are used to model the five instances of the concept. The two other neural network representing “hurt” and “sick/ill” have the same number of layers. “Hurt” has one hidden neuron and two input neurons, “sick/ill” has two hidden neurons and four input neurons.The weights of links between the output and hidden layer depend on the number of hidden neurons. The weights of links between a hidden neuron also depend on the number of its attached input neurons. Details of calculation of those weights can be found in [23]. If n instances are matched, the weight of each instance is the square root of n.During the classification phase, similarities between an input sample and each con-cepts are calculated in order to assign a concept to the sample. Classes of sounds are extracted. If an instance is found, its respective concept properties input is set to 1, otherwise it is zero. According to detected instances and their classes, weights of links of layers are determined. The output of each neural network therefore is a func-tion of weighted properties of the respective concept and is the similarity between the sample and the concept. The input sample will be assigned to the concept with which it has the highest similarity.4 Experimental EvaluationAn audio scene database of situations of distress is difficult to build. Firstly, such a database can be collected in hospitals, but recording this type of audio scenes in hos-pitals is nearly impossible due to the concerns of privacy. Secondly, recording them in a studio is feasible, but situations of distress are hard to be simulated by speakers, making a not true corpus. And finally in fact audio scenes are so numerous that it is hard to build a database (from a sound effect database) that can cover many situa-tions. Therefore we collect manually audio scenes from films. The scenes we target are the ones in which the character is in house and in a situation of distress: falling down, being hurt, sick or ill. From 150 films, 44 scenes of situations of distress with total duration of 680 seconds are collected.This classifier (illustrated in Fig. 4) should function automatically: detects sounds, estimates similarities between the sample and ontologies, and outputs the concept of the scene. But in this first stage of building an ontology-based classification system, the first steps are manually carried out. 44 audio scenes of situations of distress and50 of normal (non distress) situations are tested. Results of the ontology-based classi-fication are presented in Table 2.Fig. 4. The ontology-base classifier. Detected sounds are classified by the sound classifiers. Itsoutput is fed to the audio scene classifierThe misidentified scene of “fall” is the one in which the character falls down un-consciously making the door shut, and the unique sound we get is a shutting door. Itseems to need a redefinition for this concept. But if we add this type of sound into theontology, the system will identify actions of shutting doors as “fall”, and that willlead to wrong classification results. Therefore the fall ontology does not need to beredefined. There is one normal situation that is wrongly identified as a “hurt” one. Inthis situation the character cries when she is too happy. So arises the need of an algo-rithm (should be developed in the future) that is capable of discriminating betweenthe cries coming from happiness and those coming from hurt. Based on results ofclassification, it can be said that the ontologies are appropriate for audio scenes ofsituations of distress.Table 2. Results of ontology-based classification. Number of correctly identified is the numberof audio scenes of situations of distress which are correctly identified as its assigned concept inthe database. Number of wrongly identified is the number of audio scenes of normal situationswhich are identified as a situation of distressHurtSick/ill FallNumber of scene 20 19 5Number of correctly identified 19 19 5Number of wrongly identified 0 1 05 Conclusion and Future WorksWe present in this article an ontology-based audio scene classifier which can be usedto detect audio scenes of situations of distress. The construction of the ontology iswithin the framework of a telemedicine project. Three ontologies of sounds are de-fined. Concept, properties and instances of an ontology are modeled by a feed for-ward neural network. The output of a neural network presenting a concept is the simi-larity between it and the input sample. At first stages of developing the classifier, we defined three domain ontologies and tested them manually. These ontologies work well with our first audio database of situations of distress which is extracted from scenes of films.In the future a fully automatic ontology-based system needs to be built. In order to archieve this, the following tasks must be undertaken. First, more sound classes should be classified to cover a larger range of different types of sound. Second, sounds need to be separated from music because audio scenes collected from film are often mixed with music, making sound classifiers work inexactly. Third a combina-tion of ontologies and sound classifier should be built. Fourth more situations of dis-tress need to be defined. And finally, a bigger database should be acquired to obtain more complete domain ontologies. Besides the extension of this audio database, we also think of acquiring a text database from the Internet. This text database will con-sist of paragraphs that use types of sound in order to describe audio scenes in house. In short it is a text database of audio scenes. Audio object classes, context of the scene, or distribution of audio object can probably be extracted from this database. 6. AcknowledgementsThe authors gratefully acknowledge the receipt of a grant from the Flemish Interuni-versity Council for University Development cooperation (VLIR UOS) which enabled them to carry out this work.The authors also gratefully acknowledge the receipt of grants from French projects RESIDE-HIS and MAE/CORUS.References1. Amatriain, X., Herrera, P.: Transmitting Audio Contents as Sound Objects. Proceedings ofAES 22th International Conference on Virtual, Synthetic and Entertainment Audio, Espoo Finland (2002)2. Breen, C., Khan, L., Kumar, A., Wang, L.: Ontology-Based Image Classification UsingNeural Networks. Proc. SPIE (the International Society for Optical Engineering) Vol. 4862 (2002) 198-2083. Cano, P., Koppenberger, M., Celma, O., Herrera, P., Tarasov, V.: Sound Effects TaxonomyManagement in Production Environments. Proceedings of AES 25th International Confer-ence, London UK (2004)4. Carey, M.J., Parris, E.S., Lloyd-Thomas, H.: A Comparision of Features for Speech, MusicDiscrimination. Proceedings of the International Conference on Acoustics, Speech and Sig-nal Processing, Munich Germany (1997)5. Casey, M.: MPEG-7 Sound-Recognition Tools. IEEE Transaction on Circuits and Systemsfor Video Technology, Vol. 11, No. 6 (2001)6. Dey, L., Abulaish, M.: Ontology Enhancement for Including Newly Acquired Knowledgeabout Concept Descriptions and Answering Imprecise Queries. Web Semantics Ontology, Idea Group (2006) 189 - 2257. Gene Ontology Home: /8. Gruber, T.R.: A Translation Approach to Portable Ontology Specifications. KnowledgeAcquisition 5(2) (1993) 199-2209. Hatala, M., Kalantari, L., Wakkary, R., Newby, K, : Ontology and Rule Based Retrieval ofSound Objects in Augmented Audio Reality System for Museum Visitors. Proceedings of the 2004 ACM Symposium on Applied Computing, Nicosia Cyprus (2004) 1045 – 1050 10. Istrate, D., Vacher, M., Castelli, E., Sérignat, J.F.: Distress Situation Identifcation thoughSound Processing. An Application to Medical Telemonitoring. European Conference on Computational Biology, Paris (2003)11. Istrate, D., Vacher, M., Serignat, J.F., Besacier, J.F., Castelli, E.: Système de Télésurveil-lance Sonore pour la Détection de Situation de Détresse (Sound Telesurveillance System for Distress Situations Detection). ITBM-RBM Elsevier (2006)12. Istrate, D., Castelli, E., Vacher, M., Besacier, L., Serignat, J.F.: Sound Detection and Rec-ognition in Medical Telemonitoring. IEEE Transactions on Information Technology in Bio-medicine (to be published)13. Khan, L., McLeod, D.: Audio Structuring and Personalized Retrieval Using Ontologies.Proceedings of IEEE Advances in Digital Libraries, Library of Congress, Washington DC (2000) 116-12614. Kim, H.G., Moreau, N., Sikora, T.: Audio Classification Based on MPEG-7 Spectral BasisRepresentations. IEEE Transactions on Circuits and Systems for Video Technology, Vol.14, No. 5 (2004)15. Laegreid, A., Hvidsten, T.R., Midelfart, H., Komorowski, J., Sandvik, A.K.: PredictingGene Ontology Biological Process from Temporal Gene Expression Patterns. Genome Res13 (2003) 965-97916. Lu, L., Jiang, H., Zhang, H.J.: A Robust Audio Classification and Segmentation Method.ACM Multimedia (2001) 203-21117. Maillot, N., Thonnat, M., Hudelot, C.: Ontology Based Object Learning and Recognition:Application to Image Retrieval. In Proceedings of the 16th IEEE International Conference on Tools with Artificial Intelligence, Boca Raton FL USA (2004)18. Mezaris, V., Kompatsiaris, I., Strintzis, M.G.: An Ontology Approach to Object-BasedImage Retrieval. In proceedings of the IEEE International Conference on Image Processing (2003)19. Nakatani, T., Okuno, H.G.: Sound Ontology for Computational Auditory Scene Analysis.Proceedings of the fifteenth National Conference on Artificial Intelligence (AAAI-98), Vol.~1 (1998) 30-3520. Nguyen, C.P., Pham, N.Y., Castelli, E.: Toward a Sound Analysis System for Telemedi-cine. 2nd International Conference on Fuzzy Systems and Knowledge Discovery, Chansha China (2005)21. Noh, S., Seo, H., Choi, J., Choi, K., Jung, G.: Classifying Web Pages Using AdaptiveOntology. In Proceedings of the IEEE International Conference on Systems, Man and Cy-bernetics, Washington D.C. (2003) 2144-214922. Pfeiffer, S., Fischer, S., Effelsberg, W.: Automatic Audio Content Analysis. IEEE Multi-media, 3(3) (1996) 27-3623. Prabowo, R., Jackson, M., Burden, P., Knoell, H.D.: Ontology-Based Automatic Classifica-tion for the Web Pages: Design, Implimentation and Evaluation. The 3rd International Con-ference on Web Information Systems Engineering, Singapore (2002)。