EXAMPLE-BASED CORRECTION OF WORD SEGMENTATION AND PART OF SPEECH LABELLING

Statistical algorithms rely on frequency of phenomena or events in corpora; however, low frequency events are often inadequately represented. Here we report on an examplebased technique used in finding word segments and their part of speech in Japanese text. Rather than using hand-crafted rules, the algorithm employs example data, drawing generalizations during training.
227
2. ARCHITECTURE
The architecture in Figure 1 was chosen to minimize labor and to maximize use of existing software. It employs JUMAN first to provide initial word segmentation of the text, an annotation-based algorithm second to correct both segmentation errors and part of speech errors in JUMAN output, and POST third both to select among ambiguous alternative segmentations/partof-speech assignments and also to predict the part of speech of unkcribes an example-based correction component for Japanese word segmentation and part of speech labelling (AMED), and a way of combining it with a pre-existing rule-based Japanese morphological analyzer and a probabilistic part of speech tagger.
E X A M P L E - B A S E D C O R R E C T I O N OF W O R D S E G M E N T A T I O N A N D P A R T OF S P E E C H L A B E L L I N G
Tomoyoshi Matsukawa , Scott Miller, and Ralph Weischedel
1. INTRODUCTION
Probabilistic part of speech taggers have proven to be successful in English part of speech labelling [Church 1988; DeRose, 1988; de Marcken, 1990; Meteer, et. al. 1991, etc.]. Such stochastic models perform very well given adequate amounts of training data representative of operational data. Instead of merely stating what is possible, as a non-stochastic rule-based model does, probabilistic models predict the likelihood of an event. In determining the part of speech of a highly ambiguous word in context or in determining the part of speech of an unknown word, they have proven quite effective for English. By contrast, rule-based morphological analyzers employing a hand-crafted lexicon and a hand-crafted connectivity matrix are the traditional approach to Japanese word segmentation and part of speech labelling [Aizawa and Ebara 1973]. Such algorithms have already achieved 90-95% accuracy in word segmentation and 9095% accuracy in part-of-speech labelling (given correct word segmentation). The potential advantage of a rulebased approach is the ability of a human coding rules that cover events that are rare, and therefore may be inadequately represented in most training sets. Furthermore, it is commonly assumed that large training sets are not required.
- A probabilistic part-of-speech tagger (POST) [Meteer, et al., 1991] which assumed a single sequence of words as input.
-
Limited human resources for creating training data.
BBN Systems and Technologies 70 Fawcett St. Cambridge, M A 02138
A
B
S
T
R
A
C
T
A third approach combines a rule-based part of speech tagger with a set of correction templates automatically derived from a training corpus [Brill 1992]. We faced the challenge of processing Japanese text, where neither spaces nor any other delimiters mark the beginning and end of words. We had at our disposal the following: A rule-based Japanese morphological processor (JUMAN) from Kyoto University. A context-free grammar of Japanese based on part of speech labels distinct from those produced by JUMAN.
This presented us with four issues: 1) how to reduce the cost of modifying the rule-based morphological analyzer to produce the parts of speech needed by the grammar, 2) how to apply probabilistic modeling to Japanese, e.g., to improve accuracy to -97%, which is typical of results in English, 3) how to deal with unknown words, where JUMAN typically makes no prediction regarding part of speech, and 4) how to estimate probabilities for low frequency phenomena. Here we report on an example-based technique for correcting systemmatic errors in word segmentation and part of speech labelling in Japanese text. Rather than using handcrafted rules, the algorithm employs example data, drawing generalizations during training. In motivation, it is similar to one of the goals of Brill (1992).
I JUMAN [
Segment Correction Model
Part-of-speecl~ Model
i
Word segments with Part of Speech
Figure 1: Architecture Let us briefly review each component. JUMAN, available from Kyoto University makes segmentation decisions and part of speech assignments to Japanese text. To do this, it employs a lexicon of roughly 40,000 words, including their parts of speech. Where alternative segmentations are possible, the connectivity matrix eliminates some possibilities, since it states what parts of speech may follow a given part of speech. Where the connectivity matrix does not dictate a single segmentation and part of speech, generally longer words are preferred over shorter segmentations. An example JUMAN output is provided in Figure 2. The Japanese segment is given first, followed by a slash