, Franz Josef Och 2

Maximum Entropy Models for Named Entity Recognition Oliver Bender,Franz Josef Och,and Hermann NeyLehrstuhl f¨u r Informatik VI Information Sciences Institute Computer Science Department University of Southern California RWTH Aachen-University of Technology Marina del Rey,CA90292 D-52056Aachen,Germany och@ bender,ney@cs.rwth-aachen.deAbstractIn this paper,we describe a system that appliesmaximum entropy(ME)models to the task ofnamed entity recognition(NER).Starting withan annotated corpus and a set of features whichare easily obtainable for almost any language,wefirst build a baseline NE recognizer whichis then used to extract the named entities andtheir context information from additional non-annotated data.In turn,these lists are incor-porated into thefinal recognizer to further im-prove the recognition accuracy.1IntroductionIn this paper,we present an approach for extracting the named entities(NE)of natural language inputs which uses the maximum entropy(ME)framework(Berger et al.,1996).The objective can be described as follows. Given a natural input sequence we choose the NE tag sequence with the highest probability among all possible tag sequences: The argmax operation denotes the search problem,i.e. the generation of the sequence of named entities.Ac-cording to the CoNLL-2003competition,we concentrate on four types of named entities:persons(PER),locations (LOC),organizations(ORG),and names of miscellaneous entities(MISC)that do not belong to the previous three groups,e.g.[PER Clinton]’s[ORG Ballybunion]fans in-vited to[LOC Chicago].Additionally,the task requires the processing of two different languages from which only English was spec-ified before the submission deadline.Therefore,the system described avoids relying on language-dependent knowledge but instead uses a set of features which are easily obtainable for almost any language.The remainder of the paper is organized as follows:in section2,we outline the ME framework and specify the features that were used for the experiments.We describe the training and search procedure of our approach.Sec-tion3presents experimental details and shows results ob-tained on the English and German test sets.Finally,sec-tion4closes with a summary and an outlook for future work.2Maximum Entropy ModelsFor our approach,we directly factorize the posterior probability and determine the corresponding NE tag for each word of an input sequence.We assume that the decisions only depend on a limited window ofaround the current word and on the two predecessor tags.Thus,we obtain the follow-ing second-order model:mA well-founded framework for directly modeling the posterior probability is maximum en-tropy(Berger et al.,1996).In this framework,we have a set of feature functions.For each feature function,there exists a model parameter.The posterior probability can then be modeled as follows:Input SequencePreprocessingGlobal SearchPostprocessingTag SequenceFigure1:Architecture of the maximum entropy model approach.exp2.3TrainingFor training purposes,we consider the set of manually an-notated and segmented training sentences to form a singlelong sentence.As training criterion,we use the maximumclass posterior probability criterion:This corresponds to maximizing the likelihood of the MEmodel.Since the optimization criterion is convex,there isonly a single optimum and no convergence problems oc-cur.To train the model parameters we use the Gen-eralized Iterative Scaling(GIS)algorithm(Darroch andRatcliff,1972).In practice,the training procedure tends to result in anoverfitted model.To avoid overfitting,(Chen and Rosen-feld,1999)have suggested a smoothing method where aGaussian prior on the parameters is assumed.Instead ofmaximizing the probability of the training data,we nowmaximize the probability of the training data times theprior probability of the model parameters:where1The Reuters corpus was kindly provided by Reuters Lim-ited.Precision93.27%93.42MISC81.02%84.67%84.13PER91.91%Overall88.86%English test RecallLOC89.81%78.35%75.70ORG76.16%89.77%88.8184.68%83.92Precision72.23%71.67MISC44.95%71.90%63.27PER68.59%Overall61.16%German test RecallLOC69.66%66.52%54.58ORG53.17%87.91%81.2274.82%68.88Table1:Results of thefinal system on the developmentand test sets in English and German.Precision90.01%89.26English test82.90%73.60%64.70German test60.74%projects/muc/.S.Chen and R.Rosenfeld.1999.A gaussian priorfor smoothing maximum entropy models.TechnicalReport CMUCS-99-108,Carnegie Mellon University,Pittsburgh,PA.J.N.Darroch and D.Ratcliff.1972.Generalized iter-ative scaling for log-linear models.Annals of Mathe-matical Statistics,43:1470–1480.。