Using People and WordNet to Measure Semantic Relatedness

Using People and WordNet to MeasureSemantic RelatednessBeata Beigman KlebanovApril16,2006AbstractThis technical report describes in some detail(1)the creation ofa dataset for testing the degree of relatedness between concepts outof the data from Beigman Klebanov and Shamir’s lexical cohesionexperiment[3,5,6],and(2)a new measure of semantic relatednessbased on WordNet.We welcome comments on this manuscript;how-ever,please refrain from citing it,but rather the concise publishedversion[4].This report is intended to accompany the published pa-per with more thorough technical detail to enable replication of themethod.1IntroductionEstimating the degree of semantic relatedness between words in a text is deemed important in numerous applications:word-sense disambiguation[1], story segmentation[20],error correction[12],summarization[2,9].Furthermore,Budanitsky and Hirst(2006)noted that various applica-tions tend to pick the same measures of relatedness,which suggests a certain commonality in what is required from such a measure by the different ap-plications.It thus seems worthwhile to develop such measures intrinsically, before putting them to application-based utility tests.The most popular,by-now-standard testbed is Rubenstein and Goode-nough’s(1965)list of65noun pairs,ranked by similarity of meaning.A 30-pair subset(henceforth,MC)passed a number of replications[15,19], and is thus highly reliable.1Rubenstein and Goodenough view similarity of meaning as degree of sy-nonymy.Researchers have long recognized,however,that synonymy is only one kind of semantic affinity between words in a text[10],and expressed a wish for a dataset for testing a more general notion of semantic relatedness.1 This paper proposes and explores a new relatedness dataset.In sections2-3,we briefly introduce the experiment by Beigman Klebanov and Shamir (henceforth,BS),and show how to use it to induce relatedness scores,in-cluding estimation of their subjects’homogeneity with respect to the scores.In section4,we propose a new WordNet-based measure of relatedness, and use it to explore the new dataset.We show that it usually does better than competing WordNet-based measures(sections5,6).We discuss future directions in section7.2DataAiming at reader-based exploration of lexical cohesion in texts,Beigman Kle-banov and Shamir conducted an experiment with22students,each reading 10texts:3news stories,4journalistic and3fiction pieces[6].People were instructed to read the textfirst,and then go over a separately attached list of words in order of their appearance in the text,and ask themselves,for every newly mentioned concept,“which previously mentioned concepts help the easy accommodation of the current concept into the evolving story,if indeed it is easily accommodated,based on the common knowledge as perceived by the annotator”[5];this preceding helper concept is called an anchor.People were asked to mark all anchoring relations they couldfind.The rendering of relatedness between two concepts is not tied to any specific lexical relation,but rather to common-sense knowledge,which,ac-cording to[11](cited in the guidelines),has to do with“knowledge of kinds, of associations,of typical situations,and even typical utterances”.The phe-nomenon is thus clearly construed as much broader than degree-of-synonymy.Analyzing the experimental data,Beigman Klebanov and Shamir[3,6] exclude2outliers,and provide reliability estimation using statistical analysis. They show that items that are given an anchor by>12people are reliably anchored in the text;items left bare by all20people are reliably un-anchored.1“...there is at present no large dataset of human judgments of semantic related-ness”[12];“To our knowledge,no datasets are available for validating the results of se-mantic relatedness metric”[8].2Strong anchors for the reliably anchored items are identified and validated experimentally.Their analysis provides high-validity data for classification; however,much of the data regarding intermediate degrees of relatedness is left out.3Relatedness ScoresOur idea is to induce scores for pairs of anchored items with their anchors (henceforth,AApairs)using the cumulative annotations by20people.Thus, AApairs written by all20people would score20,and those written by just one person score1.The scores would correspond to the perceived relatedness of the pair of concepts in the given text.3.1Multi-Word ItemsIn the core data for classification[3,6],no distinctions were retained between pairs marked by19or13people,as long as the number of people passed the established threshold.Now,however,we are interested in the relative relatedness,so it is important to handle cases where the BS data might under-rate a given pair.One such case are multi-word(henceforth,complex)anchored items;the instructions asked for judgements on the given wordlist,but some subjects added complex items they perceived as a unit,and gave them anchors.Those are most often proper names,like yasser arafat/palestinian,or phrasal verbs,like passed away/died.Such items received an unsystematic treat-ment,some people inserting complex anchored items,others anchoring one part of the item in the other(arafat/yasser,away/passed),and yet others skipping the issue altogether.To exclude such cases,we remove all AA-pairs with a complex anchored item;we make a note of all complex an-chored items produced by at least2people,and remove AApairs that have parts of those complex constructs as anchored items(like yasser/palestinian, passed/died),fearing that such annotations pertain to the complex item rather than to its parts(a particularly telling example is bank/ramallah,in a text mentioning the West Bank).Another issue concerns complex anchors to single-word anchored items. The instructions ask for the minimal word sequence that is sufficient for3anchoring to occur.2However,the data shows that such minimality decision is not always straightforward:as anchors for aged,people produced old, 69years,years,69years old.The number of“votes”each version received is small,possibly undermining their actual degree of relatedness.Thus,for a given anchored item,we exclude AApairs with anchors that contain words in common with other anchors of the same anchored item;in this case,all the mentioned anchors for aged will be excluded.Of the remaining data,we retain only pairs where the anchored item and the anchor belong to open-class parts of speech;this is based on the obser-vation that functional categories contribute little to the lexical texture[10].The Size column of table1shows the sizes of the datasets for all BS texts, after the aforementioned exclusions.3.2Group Homogeneity EstimationThe induced scores correspond to the cumulative judgements of a certain group of people.How well do they represent the people’s ideas?One way to measure group homogeneity is leave-one-out estimation,as performed by Resnik(1995)for his experiment on Miller&Charles noun pairs,attaining the high average correlation of r=0.88.In the current case,however,every specific person made a binary decision,whereas a group is represented by scores1to20;such difference in granularity is problematic for correlation or rank order analysis.Another way to measure group homogeneity is to split it into subgroups and compare ranks emerging from the different subgroups.We know from Beigman Klebanov and Shamir[3,6]that it is not the case that the20-subject group clusters into subgroups that systematically produced different patterns of answers.This leads us to expect relative lack of sensitivity to the exact splits into subgroups.To validate this reasoning,we performed100random choices of two9-subject3groups,calculated the ranks induced on the AApairs by the two 2“...prefer University over The Hebrew University of Jerusalem as an anchor for a subsequent lecturer”p.5in Beigman Klebanov and Shamir(2005).3To use correlation statistics faithfully,we must make sure that there is no dependence between the judgments of the two groups.In a10/10split,there is a certain amount of dependence:if one group gives a rank of0to an AApair(none of the group members marked it),it is not possible for the second group to mark this pair0,since a pair that nobody marked would not have been included in the dataset.Choosing9/9split allows4groups,and computed Pearson correlation between the two lists.4 Thus,for every BS text,we have a distribution of100coefficients,which is approximately normal.Estimations ofµandσof these distributions are µ=.69toµ=.82,σ=.02toσ=.03for the different BS texts.Theµ(r) column in table1lists thefigures for each of the BS texts.To summarize:although the homogeneity is lower than for MC data,we observe good average inter-group correlations with little deviation across the 100splits.We now turn to discussion of a relatedness measure,which we will evaluate using the data.4GIC:WordNet-based MeasureMeasures using WordNet taxonomy are state-of-the-art in capturing semantic similarity,attaining r=.85–.89correlations with the MC dataset[13,7]. However,they fall short of measuring relatedness,as,operating within a single-POS taxonomy,they cannot meaningfully compare kill to death.This is a major limitation with respect to BS data,where only about40%of pairs are nominal,and less than10%are verbal.We develop a WordNet-based measure that would allow cross-POS comparisons,using WordNet glosses in addition to the taxonomy.One family of WordNet measures are methods based on estimation of in-formation content(henceforth,IC)of concepts,as proposed in[19].Resnik’s key idea in corpus-based information content induction using a taxonomy is to count every appearance of a concept as mentions of all its hypernyms as well.This way,artifact#n#1,although rarely mentioned explicitly,receives high frequency and low IC value.We will count a concept’s mention towards all its hypernyms AND all words5that appear in its own and its hypernyms’glosses.Analogously to artifact,we expect properties mentioned in glosses of more general concepts to be less informative,as those pertain to more things (ex.,visible,a property of anything that is-a physical object).The details both comparable scale(0-9in each),and full freedom of choice on any AApair.4We chose Pearson correlation over Spearman’s rank order correlation since the datasets contain many tied ranks,and rank order correlation tends to show higher-than-real values in such cases.5We induce IC values on(POS-tagged base form)words rather than senses.Ongoing gloss sense-tagging projects like eXtended WordNet (/links.html)would allow sense-based calculation in the future.5of the algorithm for information content induction from taxonomy and gloss information(IC GT)are given in appendix A.To estimate the semantic affinity between two senses A and B,we average the IC GT values of the3words with the highest IC GT in the overlap of A’s and B’s expanded glosses(the expansion follows the algorithm in appendix A).6 If A∗(the word of which A is a sense)appears in the expanded gloss of B,we the maximum between the IC GT(A∗)and the value returned by the 3-smoothed calculation.To compare two words,we take the maximum value returned by pairwise comparisons of their WordNet senses.7The performance of this measure is shown under GIC in table1.GIC manages robust but weak correlations,never reaching the r=.40threshold. 5Previous Work:RelatednessWe compare GIC to another WordNet based measure that can handle cross-POS comparisons,proposed by Banerjee and Pedersen(2003).To compare word senses A and B,the algorithm compares not only their glosses,but also glosses of items standing in various WordNet relations with A and B.For example,it compares the gloss of A’s meronym to that of B’s hyponym.We use the default configuration of the measure in WordNet::Similarity-0.12package[16],and,with a single exception,the measure performed below GIC;see column BP in table1.Experimenting with this measure,we tried applying to it the following transformation.Suppose BP is the score returned by the original BP algo-rithm on a certain pair;then:EBP=e BP−e−1×BP e BP+e−1×BPThe transformation was proposed in[14],and applied to linear mea-sures used therein.If,for the original measure,0means no similarity,and scores close and above1mean much similarity,the transformation scales the 6The number3is empirically-based;the idea is to counter-balance(a)the effect of an accidental match of a single word which is relatively rarely used in glosses;(b)the multitude of low-IC items in many of the overlaps that tend to downplay the impact of the few higher-IC members of the overlap.7To speed the processing up,we usefirst5WordNet senses of each item for results reported here.6Dataset Sizeµ(r)Gic BP EBPBS-11007.71.29.19.19BS-2776.79.37.16.20BS-31015.73.22.09.11BS-4512.73.34.39.37BS-51020.75.25.11.17BS-6536.69.24.19.22BS-7917.72.22.10.11BS-8529.76.24.12.11BS-9509.76.31.16.20BS-10417.82.36.19.20Av BS724.75.28.17.19Table1:Dataset sizes,homogeneity and correlations of GIC,BP with human ratings.r>0.16is significant at p<.05;r>.23is significant at p<.01. scores such that the differences between higher scores are diminished.So, BP=0.20and BP=0.30turn into EBP=0.20and EBP=0.29,respec-tively,nearly preserving the difference.If the originals are.85and.95,the transformed ones are.69and.74,with only.05difference.Finally,had the originals been3and10,the transformed values are0.995054753686731and 0.999999995877693,hardly different at all.If used for a similarity measure, this means that a pair of very similar things should score closer to another pair of very similar things than the original measure would predict.The last column in table1shows the correlations attained by the trans-formed version EBP.We note that it usually improves slightly on the original BP measure;still,it is doing worse than Gic in all but one cases.6Previous Work:SimilarityAs mentioned before,taxonomy-based similarity measures cannot fully han-dle BS data.Table2uses nominal-only subsets of BS data and the MC nominal similarity dataset to show that(a)state-of-the-art WordNet-based similarity measure JC8[13,7]does very poorly on the relatedness data,sug-gesting that nominal similarity and relatedness are rather different things;8See appendix B.7(b)GIC does better on average,and is more robust;(c)GIC yields on MC to gain performance on BS,whereas BP is no more inclined towards relatedness than JC.Dataset Size Gic BP EBP JCMC30.78.80.83.86BS-1328.38.18.19.21BS-2249.53.18.22.37BS-3259.21.04.05.01BS-4125.28.38.33.33BS-5505.12.07.11.16BS-6286.25.16.20.22BS-7309.23.10.09.04BS-8257.32.10.08.00BS-9178.24.17.18.27BS-10186.41.25.24.25Av BS268.30.16.17.19Table2:MC and Nominal-only subsets of BS:correlations of various mea-sures with the human ratings.Table3gives an example of relatedness vs.similarity distinction.Whereas, taxonomically speaking,son is more similar to man,as reflected in JC scores, people marked family and mother as much stronger anchors for son in BS-2; GIC follows suit.AApair Human Gic JCson–man20.35522.3son–family130.37516.9son–mother160.37020.1Table3:Relatendess vs.Similarity Example87Conclusion and Future WorkWe proposed a dataset of relatedness judgements that differs from the exist-ing ones9as follows:•size:about7000items,as opposed to up to350in existing datasets;•cross-POS data,as opposed to purely nominal or verbal;•a broad approach to semantic relatedness,not focussing on any particular relation,but grounding it in the reader’s(idea of)common knowledge;this as opposed to synonymy-based similarity prevalent in existing databases.We explored the data with WordNet-based measures,showing that(1) the data is different in character from a standard similarity dataset,and very challenging for state-of-the-art methods;(2)the proposed novel WordNet-based measure of relatedness usually outperforms its competitor,as well as a state-of-the-art similarity measure when the latter applies.In future work,we plan to explore distributional methods for modeling relatedness,as well as the use of text-based information to improve corre-lations with the human data,as judgments are situated in specific textual contexts.References[1]Satanjeev Banerjee and Ted Pedersen.Extended gloss overlaps as ameasure of semantic relatedness.In Proceedings of IJCAI,2003.[2]Regina Barzilay and Michael ing lexical chains for textsummarization.In Proceedings of ACL Intelligent Scalable Text Sum-marization Workshop,1997.[3]Beata Beigman ing readers to identify lexical cohesivestructures in texts.In Proceedings of ACL Student Research Workshop, 2005.9MC is not the only available dataset;we will address other datasets in a subsequent paper.9[4]Beata Beigman Klebanov.Measuring semantic relatedness using peopleand WordNet.In Proceedings of NAACL Short Papers,2006.[5]Beata Beigman Klebanov and Eli Shamir.Guidelines for annotation ofconcept mention patterns.Technical Report2005-8,Leibniz Center for Research in Computer Science,The Hebrew University of Jerusalem, Israel,2005.[6]Beata Beigman Klebanov and Eli Shamir.Reader-based exploration oflexical nguage Resources and Evaluation,to appear,2006.Springer,Netherlands.[7]Alexander Budanitsky and Graeme Hirst.Evaluating WordNet-basedmeasures of semantic putational Linguistics,32(1):13–47, 2006.[8]Iryna ing the structure of a conceptual network in com-puting semantic relatedness.In Proceedings of IJCNLP,2005.[9]Iryna Gurevych and Michael Strube.Semantic similarity applied tospoken dialogue summarization.In Proceedings of COLING,2004. [10]M.A.K.Halliday and Ruqaiya Hasan.Cohesion in English.LongmanGroup Ltd.,1976.[11]Graeme Hirst.Context as a spurious concept.In Proceedings of CI-CLING,2000.[12]Graeme Hirst and Alexander Budanitsky.Correcting real-word spellingerrors by restoring lexical cohesion.Natural Language Engineering, 11(1):87–111,2005.[13]Jay Jiang and David Conrath.Semantic similarity based on corpusstatistics and lexical taxonomy.In Proceedings on International Confer-ence on Research in Computational Linguistics,1997.[14]Yuhua Li,Zygaur Bandar,and David McLean.An approach to mea-suring semantic similarity between words using multiple information sources.IEEE Transactions on Knowledge and Data Engineering, 15(4):871–882,2003.10[15]George Miller and Walter Charles.Contextual correlates of semanticnguage and Cognitive Processes,6(1):1–28,1991.[16]Ted Pedersen,Siddharth Patwardhan,and Jason Michelizzi.WordNet::Similarity-measuring the relatedness of concepts.In Proceedings of NAACL,2004.[17]Adwait Ratnaparkhi.A maximum entropy model for part-of-speechtagging.In Proceedings of EMNLP,1996.[18]Jason Rennie.WordNet::QueryData:a Perl module for accessingthe WordNet database./~jrennie/WordNet, 2000.[19]Philip ing information content to evaluate semantic similarityin a taxonomy.In Proceedings of IJCAI,1995.[20]Nicola Stokes,Joe Carthy,and Alan F.Smeaton.SeLeCT:A lexicalcohesion based news story segmentation system.Journal of AI Com-munications,17(1):3–12,2004.A Gloss&Taxonomy IC(IC GT)We now detail the method for IC induction from taxonomy and gloss infor-mation.First,we part-of-speech-tag[17]and bring to the base form all words in WordNet glosses10;we refer to such items when we say“words”through-out this section.Then,for every word-sense W in WordNet database for a given part of speech:1.Collect all content words from the gloss of W,excluding examples,including W∗-the part-of-speech-tagged word of which W is a sense.2.If W is part of a taxonomy,expand its gloss,without repetitions,withwords appearing in the glosses of all its super-ordinate concepts,up to the top of the hierarchy.Thus,the expanded gloss for airplane#n#1 would contain words from the glosses of the relevant senses of aircraft ,vehicle,transport,etc.The idea behind this step is that mention of 10We use WordNet::QueryData[18]package version1.38to access WordNet-2.0.11properties in a taxonomic database is often organized in an inheritance hierarchy-a property is mentioned at the most general level it applies.Thus,the rather salientflying property of airplane pertains already to its super-super-ordinate,aircraft.3.Add sense count of W to all words in its expanded gloss.11Each part-of-speech database induces its own counts on each word that appeared in the gloss of at least one of its bining raw counts would effectively correspond to the data in the nominal database,by far the largest.While acceptable in general,it downplays the fact that,adverbs,for example,often describe the manner in which some action is performed.Thus, the concept of manner would be relatively un-informative as far as adverbs are concerned,whereas it would seem more informative from the point of view of nominal definitions.Hence,when merging the information from the different parts of speech,we scale the aggregated counts,such that they correspond to the proportion of the given word in the part-of-speech database where it was the least informative.The standard log-frequency calculation transforms these counts into taxonomy-and-gloss based information content (IC GT)values.B JC measure of similarityIn the formula below,IC is taxonomy-only based information content,as articulated in[19],LS is the lowest common subsumer of the two concepts in the WordNet hierarchy,and Max is the maximum distance between any two concepts in the given hierarchy.For two concepts c1and c2:JC(c1,c2)=Max−(IC(c1)+IC(c2)−2×IC(LS(c1,c2)) To make JC scores comparable to Gic’s[0,1]range,the score can be divided by Max.Normalization has no effect on correlations.We use[16]implementation of JC measure,with the following alteration: instead of turning distance into similarity by1/Distance,as done in[16],we follow Jiang and Conrath’s(1997)proposal and subtract it from Max–the longest possible distance between two concepts in a hierarchy.The longest 11We do add-1smoothing on WordNet sense counts.12possible distance is between two concepts with highest possible IC(a single appearance count)that share only the most general concept(topmost level of the nominal hierarchy);this value is about26for WordNet-2.0and the add-1smoothed SemCor database.13。