Developing a test collection for biomedical word sense disambiguation

Developing a Test Collection for Biomedical Word Sense Disambiguation Marc Weeber,PhD,James G.Mork,MSc,Alan R.Aronson,PhD{weeber,mork,alan}@Lister Hill National Center for Biomedical CommunicationsNational Library of Medicine8600Rockville PikeBethesda,MD20894Ambiguity,the phenomenon that a word has more than one sense poses difficulties for many current Natural Language Processing(NLP)systems.Algorithms that assist in the resolution of these ambiguities,i.e.disam-biguate a word,or more generally,a text string,will boost performance of these systems.To test such tech-niques in the biomedical language domain,there is the need for a test collection of disambiguated ambigu-ous strings.We report on the development of a Word Sense Disambiguation(WSD)test collection that com-prizes5,000disambiguated instances for50ambigu-ous UMLS®Metathesaurus®strings.INTRODUCTIONConsider the following sentences that include the word cold taken from three different MEDLINE®ab-stracts:1(1)A greater proportion of mesophil micro-organisms were to be found during the coldmonths than in warmer months.(2)In a controlled randomised trial we analysedwhether the use of the term“smoker’s lung”instead of chronic bronchitis when talking topatients with chronic obstructive lung disease(COLD)changed their smoking habits.(3)The overall infection rate was83%and of thoseinfected,88%felt that they had a cold.The sense of the word cold is different in each sen-tence.Cold in sentence(1)is an indication of the tem-perature,in sentence(2)the acronym of chronic ob-structive lung disease and in sentence(3)cold is a dis-ease.The fact that a single word may have more than one sense is called ambiguity.In natural language,am-biguity occurs at many levels,e.g.,lexical,structural, semantic,and pragmatic.Also,it pervades normal lan-guage use;humans have to disambiguate constantly (and subconsciously)in normal communication using textual and other types of context.The general opinion is that language in more restricted environments,such as medical research,is more spe-cific and straightforward;there is less ambiguity.This may well be the case,but ambiguity is still present as shown by the examples above.Additionally,the UMLS®Metathesaurus®[2],the largest medical the-saurus,has more than7,400ambiguous strings that map to more than one thesaurus concept[3].The word cold,for instance,maps to six different UMLS con-cepts,three of which we used in sentences(1)–(3).MEDICAL NLP AND AMBIGUITY Medical NLP systems,generally designed to analyze medical texts for decision support or indexing pur-poses,have to deal with ambiguities in language. Columbia University’s MedLEE system,originally de-signed for a small medical(and language)domain has been applied to differentfields within medicine. One of the problems encountered when broadening the scope of such a system is the introduction of ambigu-ities.A term or word has different senses in different medical disciplines.MedLEE has some ad-hoc rules to deal with ambiguities,but there is a need for new, machine learning(ML)techniques and a good collec-tion of training data[4].The objective of the National Library of Medicine (NLM)’s Indexing Initiative is to investigate NLP methods whereby automated indexing techniques can partially or completely substitute for current(manual) indexing practices[5].Error analysis of the index-ing system shows that the major problems concern ambiguity of strings.Also,MetaMap,a text to con-cept mapping program[6,7]is currently unable to disambiguate ambiguous concepts.The DAD-system, a concept-based tool for literature-based discovery in biomedicine[8,9]uses MetaMap for the processing of MEDLINE texts.In replicating Swanson’s literature-based discovery of the involvement of magnesium de-ficiency in migraine[10],the DAD-system showed that the abbreviation mg might be interesting for treating migraine.However,the DAD-system is not able to distinguish between the UMLS concepts Magnesium1The PubMed[1]ID’s are9477717,9411973,and9578931respectively.and Milligram for mg.This means that spurious infor-mation on milligram is included in the system’s out-put[9].In their recent study on UMLS concept index-ing,Nadkarni et al.think a fully automatic procedure is not yet feasible,in part because of ambiguity prob-lems[11].Though there is clearly a need,the only research on biomedical word sense disambiguation are[12]and[4].These two studies use rule-based approaches fora few cases in small domains.Recently,WSD has seen an upsurge of interest in computational linguis-tics,illustrated by a1998special issue of Computa-tional Linguistics,V ol24(1)and a2000special issue of Computer and the Humanities,V ol.34(1/2).Ad-ditionally,there are the SENSEV AL workshops.2The time is ripe to test the newly developed algorithms in the biomedical language domain.Essential for test-ing the algorithms is a collection of manually disam-biguated biomedical text strings for use as a gold stan-dard.This paper reports on the development of such a WSD test collection.EXTENT OF AMBIGUITY IN MEDLINE To appreciate the amount of ambiguity present in MEDLINE,we processed the409,337citations added to the citation database in1998.The processing con-sisted offinding UMLS concepts in the titles and ab-stracts of these citations by means of the MetaMap program.MetaMap chunks the sentences into(mostly noun)phrases that are mapped to UMLS concepts.In this experiment,we use the1999version of the UMLS. Table1displays some basic statistics.Table1:Mapping Results for1998MEDLINE.No.of citations409,337No.of non-ambiguous phrases30,514,468No.of ambiguous phrases4,051,445We observe that11.7%of the more than34million phrases result in more than one mapping to UMLS concepts,i.e.there is an ambiguous mapping.The dif-ferences between concepts are best depicted by the dif-ferent semantic types that have been assigned to them. Studying the data,we observed three types of ambigu-ities:a)simple ambiguities in which a string maps to more than one UMLS concept(94.3%of all cases),b) lexical ambiguities(5.5%),and c)complex ambigui-ties(0.2%).See Table2for examples.Table2:Three Types of Ambiguities.Type UMLS concept Semantic typeSimple:activityActivity1FindingActivity2Daily or recr.activity%activity Quantitative concept Lexical:reportedReporting Health care activityReports Intellectual productReport2Intellectual product Complex:reproductive health policiesReproduction Organism function+Health Idea or concept+Policies Regulatory activityReproductive Health Occupation or discipline+Policies Regulatory activityReproduction Organism function+Health Policies Regulation or lawMETHODSBecause complex ambiguities are both difficult and rare,and because lexical ambiguities should be re-solved by better parsing strategies,we focus on simple ambiguities in the remainder of this paper.To disam-biguate the strings we use human raters.Selection of StringsBased on the list of ambiguous UMLS strings,we have selected50highly frequent ones for inclusion into the test collection.They are tabulated in Table3.Some highly frequent strings were not included because the concepts they are mapped to were either difficult to distinguish or the UMLS did not provide informative and consistent definitions and(hierarchical)relation-ships.The second and seventh columns provide the strings’frequency of occurrence in the1998MEDLINE ci-tations.Columns three and eight provide the num-ber of different senses,or UMLS concepts to which a string maps.For some cases,we do not use all con-cepts available in the UMLS because we judged some of them to be too close in sense to make a practical distinction.Columns4and9tabulate the number of concepts we discarded for each string.For instance, MetaMap maps the string depression to three differ-ent UMLS concepts:Depression motion,Depressive episode,unspecified,and Mental Depression.The lat-ter two concepts are very close in sense,so we decided to use only the second of the two,Mental depression,2See /senseval2/for more information.Table3:Ambiguous Strings in the NLM’s WSD Test Collection.The italicized ones are problematic to obtain a good agreement between raters.Excl R=rater excluded,and Excl S=number of senses excluded.String Occurrences Senses Excl S Excl R String Occurrences Senses Excl S Excl R adjustment2,59642lead9,8803association18,5313man5,2434blood pressure6,71341mole3,64241cold2,4486mosaic5695condition24,8913nutrition3,45641culture20,63531pathology4,37331degree17,4193pressure9,11841depression7,57731radiation5,8223determination36,7793reduction22,9793discharge5,07231repair6,77131energy7,32731resistance13,1323evaluation19,31931scale6,7344extraction10,8313secretion13,27631failure7,9893sensitivity16,1734fat6,1123sex7,2144fit3,5913single29,3113fluid5,9913strains15,8733frequency16,24431support20,2283ganglion5803surgery22,53931glucose11,2053transient7,0533growth20,7123transport10,0183 immunosuppression1,5963ultrasound5,70431 implantation4,1703variation10,4313inhibition24,1213weight12,8573japanese2,9243white4,38431since the UMLS vocabularies define this concept more clearly.For each string,we have added the sense“none”which the raters can select when none of the available senses suit a particular instance.Following the depression ex-ample,there are two UMLS senses plus the“none”op-tion which leads to an ambiguity of degree three(Ta-ble3,columns3and8).The discussion on which strings to use for the test col-lection and which senses to include for each string took place in a team of11,the authors plus eight other re-searchers at the NLM with various backgrounds in li-brary sciences,linguistics,medical informatics,and medicine.The members of this group also served as raters who disambiguated the instances.For every one of the50strings,we selected100in-stances at random from the1998MEDLINE collec-tion.Almost all of these instances originate from dif-ferent citations.Thus,there were5,000instances to be disambiguated.Disambiguation ProcedureSince disambiguating5,000instances of ambiguity manually is a non-trivial task,we developed a web-based interface that facilitates the disambiguation pro-cedure and reduces the actual manual task to two mouse clicks for each instance,see Figure1for a screenshot.The left panel of the interface presents the to be disam-biguated string in red.The sentence in which it occurs, the direct context,appears in a blue box.Additionally, the rest of the title and abstract of the MEDLINE cita-tion is visible.The raters were permitted to address the strings in any order and were not required to complete a string before starting another.The order in which the100instances for every string were presented had been randomized for every user.The different con-cepts(senses)are available in the right panel.The rater can only select one concept(radio button)or pass the instance to reconsider it at a later moment in time. Concepts and their semantic types have hyperlinks to the UMLS.Analysis of RatingsTo reach afinal classification on the correct sense, there are two approaches.Thefirst one is majority vot-ing.The sense that is selected by most raters will be thefinal and correct sense.The second method is latent class analysis(LCA)[13,14].This statistical method tries tofind the underlying and“true”classifications.Figure1:Disambiguation User Interface.The left panel shows the MEDLINE citation as context to the raters to disambiguate the string cold.The possible senses(concepts),with hyperlinks to the UMLS,are in the right panel.This method may especially be useful when majority voting results in a tie.For any particular instance,LCA uses the rating patterns of the other instances to decide which is the true andfinal classification.In addition to these methods,it may be interesting tofind out to what extent raters agree and disagree with each other using the kappa()statistic[15].The determination of thefinal classification is a four-step process.We repeated this process for all50 strings.During step one,we compute the statistic for each rater–rater combination.This statistic shows which raters agree with each other,and more impor-tantly,which raters disagree systematically from all others.We use the latter information in step two.In step two,we count the total ratings for each instance of the string.If there is a majority of two votes for a certain sense,this will be thefinal classification.In case of ties,or many majorities of one,it may be inter-esting to exclude a rater if this rater disagrees system-atically with all the others.We apply step three if step two does not result in sat-isfactory results for many instances of the string,i.e. there are many ties and majorities of one and exclud-ing one(or more)raters does not improve results.For these cases,we use LCA to obtain a classification. Step four is the reassessment of instances in a group discussion of the disambiguation team.These in-stances did not obtain a reliable classification by step 2or step3.RESULTSDepending on the difficulty of the case,raters spent between thirty minutes and two hours per ambiguous string(100instances).The rating task was done in addition to the raters’normal tasks.After a period of four months,during which there were three meet-ings in which the group discussed examples of dif-ficult strings and particular instances,the data were frozen.Eight raters completed all the5,000instances, the other three completed2800(28strings),2200(22 strings),and600(6stings)respectively.The agreement analysis by the statistic provided many interesting insights.For instance,the two raters who agreed best for most of the50strings are both for-mer NLM indexers(the only two in the team).Also, for many strings,one or two of the raters disagreed systematically with the rest of the group.By exclud-ing them in eleven cases(columns5and10in Table3) we are able to resolve ties and many majorities of one. Eight raters were excluded at least once.Steps1and 2were sufficient for38strings.Only162of the3,800 instances had to be discussed in the team for afinal classification(step4).The twelve remaining strings, written in italics in Table3,were more problematic in that there are many ties and majorities of one.Afterusing LCA,still159of the1,200instances had to be discussed in the group to reach afinal classification.DISCUSSIONAt the National Library of Medicine,we have devel-oped a test collection for word sense disambiguation research.This collection will hopefully prove valuable for the future developments of medical NLP tools.As afirst step we will apply different machine learning al-gorithms to disambiguate a string based on its context. The definition of the context will be one of the major challenges.The test collection provides the PubMed ID,the sentence in which the string occurs,the syn-tactic tags of the words in the sentence and the con-cepts that are found in the sentences by MetaMap[7]. Included with the concepts are their semantic types, therefore the semantic context may be included in the feature list that can be used by the algorithms.We observe a distinction between two type of strings in the test collection:normal and problematic ones.For the problematic ones,it was difficult to obtain agree-ment among the raters on which sense is the accurate disambiguation of many of a string’s instances.When human judgment is problematic,it may be impossible to automate disambiguation reliably.We therefore rec-ommend tofirst consider the38normal strings(3,800 instances)with ML algorithms before turning to the problematic ones.By Summer2001,The WSD test collection will be available as a UMLS resource from the NLM at /.ACKNOWLEDGMENTSWe would like to express our gratitude to the members of the disambiguation team:Florence Chang,Cliff Gay,Mike Hazard,Susanne Humphrey,Tom Rind-flesch,Sonya Shooshan,Carolyn Tilley,and Sara Ty-baert.This research was supported in part by an appointment to the NLM Research Participation Program adminis-tered by the Oak Ridge Institute for Science and Edu-cation through an interagency agreement between the U.S.Department of Energy and the NLM.REFERENCES[1]National Library of Medicine.PubMed,2001./ entrez/query.fcgi?db=PubMed.[2]National Library of Medicine.Unified medi-cal language system knowledge sources,2001./.[3]Roth L,Hole WT.Managing name ambiguity inthe UMLS metathesaurus.In:Proc AMIA AnnuFall Symp2000.Philadelphia,PA:Hanley andBelfus,2000;p.1124.[4]Friedman C.A broad-coverage natural languageprocessing system.In:Proc AMIA Annu FallSymp2000.Philadelphia,PA:Hanley and Bel-fus,2000;pp.270–274.[5]Aronson AR,Bodenreider O,Chang HF,Humphrey SM,Mork JG,Nelson SJ,et al.TheNLM indexing initiative.In:Proc AMIA AnnuFall Symp2000.Philadelphia,PA:Hanley andBelfus,2000;pp.17–21.[6]Aronson AR.The effect of textual variation onconcept based information retrieval.In:ProcAMIA Annu Fall Symp1996.Philadelphia,PA:Hanley and Belfus,1996;pp.373–377.[7]Aronson AR.Effective mapping of biomedicaltext to the UMLS metathesaurus:The metamapprogram.In:Proc AMIA Annu Fall Symp2001.Philadelphia,PA:Hanley and Belfus,2001;inpress.[8]Weeber M,Klein H,Aronson AR,Mork JG,deJong-van den Berg LTW,V os R.Text-baseddiscovery in biomedicine:The architecture ofthe DAD-system.In:Proc AMIA Annu FallSymp2000.Philadelphia,PA:Hanley and Bel-fus,2000;pp.903–907.[9]Weeber M,V os R,Klein H,de Jong-van den Berging concepts in literature-based discov-ery:Simulating Swanson’s raynaud–fish oil andmigraine–magnesium discoveries.J Am Soc InfSci;52(7):548–557.[10]Swanson DR.Migraine and magnesium:Elevenneglected connections.Perspect Biol Med1988;31(4):526–557.[11]Nadkarni P,Chen R,Brandt C.UMLS conceptindexing for production databases.J Am MedInform Assoc2001;8(1):80–91.[12]Rindflesch TC,Aronson AR.Ambiguity res-olution while mapping free text to the UMLSMetathesaurus.In:Proc Annu Symp ComputAppl Med Care1994.Philadelphia,PA:Hanleyand Belfus,1994;pp.240–244.[13]Bruce RF,Wiebe JM.Recognizing subjectivity:A case study on manual tagging.Nat Lang Eng1999;5(2):187–205.[14]Wiebe JM,Bruce RF,O’Hara TP.Developmentand use of a gold-standard data set for subjec-tivity classifications.In:Proc ACL1999.Cam-bridge:The MIT Press,1999;pp.246–253. [15]Fleiss JL.Measuring nominal scale agree-ment among many raters.Psychol Bull1971;76(5):378–382.。