Intelligent Feedback 1 IN THE PAST FEW DECADES, COMPUTER-ASSISTED LANGUAGE Instruction (CAL

Modern Language Journal 77 (3), 330-339, 1993.Intelligent Computer Feedback for Second Language InstructionNoriko NagataDepartment of Modern and Classical LanguagesUniversity of San Francisco2130 Fulton StreetSan Francisco, CA 94117IN THE PAST FEW DECADES, COMPUTER-ASSISTED LANGUAGE Instruction (CALI), or Computer-Assisted Language Learning (CALL), has been of interest because of its potential for providing individualized instruction. Interactivity between the computer and the learner is one of the important elements in CALI. Compared to a traditional textbook or workbook, a CALI program can provide immediate feedback on the correctness of the learner's response. One obstacle, however, has been the relatively primitive state of the existing CALI systems, many of which possess only limited capabilities for analyzing learners' responses, diagnosing their errors, and providing feedback. Garrett (p. 174) addresses the problem with existing CALI programs as follows: "Most students learn no more from grammar on the computer than they do from grammar in the workbooks, because much of the commercially available offerings are of the 'wrong, try again' model, which only indicates whether student-produced bits of language match the target language model stored in computer memory (sometimes also showing the correct answer) without indicating how or why the student input does not match."If the potential of computer-assisted language instruction as individualized supervised learning is to be realized, we must develop programs that support detailed error analysis and feedback targeted to specific deficiencies in the student’s performance.Garrett describes four types of feedback in relation to error analysis in CALI grammar lessons: 1) presents only the correct answer; 2) pinpoints the location of errors on the basis of the computer's letter-by-letter comparison of the student's input with the machine-stored correct version (the so-called "pattern markup" technique); 3) based on analysis of the anticipated wrong answers, error messages associated with possible errors are stored in the computer and are presented if the student's response matches those possible errors (the so-called "error anticipation" technique); 4) most sophisticated, uses an intelligent, Natural Language Processing (NLP) approach such as the "parsing" technique in which the computer does a linguistic analysis of the student's response, comparing it to an analysis derived from the relevant grammar rules and lexicon of the target language, and identifies problematic or missing items of the student's response.Conventional CALI systems can handle feedback types up to level three, but they cannot analyze the student's response on the basis of linguistic structure. Therefore, exercises in traditional CALI typically involve very mechanical language activities, focusing on low level (word-level) processing and on one principal point of grammar at a time. Error analysis and feedback in traditional CALI tend to be superficial and fail to explain why the response is wrong. On the other hand, the goal of intelligent CALI is to provide the fourth level of feedback: to analyze the student's response linguistically, to diagnose the errors, and to respond to thepossibly many points of usage that may be involved in the student's production. This paper describes a new CALI system, Nihongo-CALI, which employs one type of artificial intelligence approach, Natural Language Processing, to facilitate more sophisticated error analysis and feedback for students of Japanese.1 The exercises in Nihongo-CALI permit the students to answer questions with full sentences, stimulating sentence-level processing skills.Clearly, the more productive the exercises a system provides, the greater variety of errors it must be able to handle. Liou states that when a program detects a grammatical error, appropriate feedback messages are essential for the grammar checker to achieve its educational goal. Liou describes a program (using a parser) designed to check the grammar in ESL students' compositions. This grammar checker can locate seven types of errors, such as determiner-noun disagreement, subject-verb disagreement, erroneous morphological changes in verb phrase (e.g., "I should went with you"), although ... but combination, and so forth. The following (Liou, p. 66) illustrates an example of feedback messages provided for an ungrammatical sentence, "The development in scientific technologies have bring great change": "The development in scientific technologies have bring great change." [development is the subject of the verb have. The subject is in 3rd person singular form. The following are 2 correct examples:The clerk beside the book shelves is watching television.The lady who the workers love teache s English.]Although at this point the coverage of errors in this program is limited to several types of errors, the program will aid students' self-correction of their grammatical mistakes, thereby also saving the instructor's time in correcting students' formal errors.Labrie and Singh have been developing MINIPROF, a program designed to provide grammar exercises which involve affirmative-negative transformation, subject-verb agreement in person, number, and gender, and inflections of verb endings. In the exercises, the learners are asked to provide negative responses to yes/no questions. The system parses the learners' responses, diagnoses their errors, and provides tutorial messages. The following (Labrie and Singh, p. 10) illustrates an example of the exercises and feedback messages provided in MINIPROF:"Est-ce que tu parles espagnol?"> Non, je ne parles espagnol.The subject is "je." "parl" is correct. What should the ending be?> eRight. Your answer is negative. What do you need after the verb?> I don't know.Use "pas." Answer: Non, je ne parle pas espagnol.This type of exercise might be considered a mechanical drill, because the original sentence is always given for transfer to the negative form and the students may drill without considering the meaning. However, MINIPROF's capability of analyzing learners' responses surely makes itpossible to provide more meaningful and communicative practice, such as asking the student to respond to an imaginary conversational partner provided by the computer.Sanders poses a question about an instructional parser: "What kind of error analysis can--or should--a parser provide to second language?" She notes that availability of syntactic parsers designed for language learners can help to answer the first part of the question (what kind of error analysis a parser can provide); the second part (what kind of error analysis a parser should provide) awaits empirical studies. The important question here is not only what intelligent CALI can do for second language instruction, but also what kind of learning task, error analysis, and feedback intelligent CALI should provide to facilitate second language acquisition. Compared to traditional CALI, intelligent CALI (using a parser) has advanced capabilities in analyzing learners' responses, diagnosing errors, and providing feedback. We need empirical data that will give us some answer to the question of why we need intelligent CALI, and what error analysis and feedback intelligent CALI should provide.Many studies have investigated the effect of immediate feedback provided by the computer. For example, O'Day and Kulhavy tested the principle of immediate feedback with a program for teaching biological terms. They compared different types of feedback, such as knowledge of correct response (KCR), KCR plus remediation (where a remedial message is given for an incorrect response), or no feedback. Their study found no statistically significant differences between different types of feedback in either posttest gain or learning efficiency. Their study did not support the principle of immediate feedback, whether this principle is interpreted to mean KCR only, remediation only, or the combined effects of both. However, the content of tasks and the quality of feedback could be questioned. Their study employed multiple-choice questions to test comprehension of biological terms. Their exercises did not require processing of complex knowledge, rule application, or creative thinking, and the remedial feedback provided by their program was quite simple as well. The issue is not simply whether or not immediate feedback brings about learning effects; the effects of immediate feedback depend on what instructional goal is posed, what skill is focused on to develop, what type of learning task is required, and what kind of error analysis and feedback is provided in the program.The present study involved the development of the Nihongo-CALI system and compared two versions of this system: an intelligent version and a more traditional one. The goals were to employ the Natural Language Processing approach in order to make possible more sophisticated error analysis and feedback, to demonstrate what kind of learning task, error analysis, and feedback Nihongo-CALI can provide, and to investigate what type of feedback intelligent CALI should provide (compared to traditional CALI) in order to develop the learners' grammatical skills and language accuracy. Natural Language Processing analyzes the processing of natural (human) languages in terms of morphology, syntax, semantics/pragmatics, and cognitive processes, and aims to formulate the processing rules of a language by which people understand and produce the language. The Natural Language Processing approach in Nihongo-CALI is based on theories of language acquisition and psycholinguistic processes of language learning that open up the possibility of providing useful feedback to the language learners. Although Natural Language Processing involves many components, such as morphology, syntax, the lexicon, semantics/pragmatics, discourse analysis, inference, etc., thepresent study focused on only three of these components: morphology, syntax, and the lexicon. The computational-linguistic issues addressed here included handling of both grammatical and ungrammatical input and providing grammatical information and error correction to the learner. The experimental focus was on the effect of the type of feedback (traditional or intelligent) on the learners' performance in producing Japanese sentences. The next section describes error analysis and feedback provided by Nihongo-CALI.ERROR ANALYSIS AND FEEDBACK IN NIHONGO-CALIThe Nihongo-CALI system employs Natural Language Processing to process and to analyze the learner's responses in light of the morphological and syntactic rules of the Japanese language. Figure I illustrates the process of analyzing a sentence by Natural Language Processing (NLP).Figure I : Natural Language ProcessingInput sentenceOutput messages (results of NLP analysis)Grammatical information about the sentenceIn this system, the learner's input sentence is sent to an NLP analyzer which consists of a morphological parser (4), a syntactic parser (19), and three kinds of knowledge bases: a core lexicon, a set of morphological rules, and a set of grammar (syntactic) rules.2 The results of NLP analysis are presented in terms of grammatical functions and thematic roles of each component of the sentence. Nihongo-CALI includes ninety exercises for practicing Japanese passive structures.3 These exercises employ a type of utilization exercise used in a Japanese textbook, Japanese: The Spoken Language (5). Japanese passive structures were the target structures involved in this study. The Nihongo-CALI program runs both on a Macintosh personal computer (with minimum of five MB of RAM and five MB of disk) and on a workstation (DECstation).4Figure II shows an example of the Nihongo-CALI exercises on the computer screen.Figure II: Format of the Nihongo-CALI ExercisesOptions1. Respond to the following conversation, according to the situation described.(1) Your colleague told you that he called your office this morning but nobody answered. Respond that yes, the secretary came in late (i.e., you were affected by the secretary's coming in late).Colleague: Kesa zimusyo ni denwa simasita kedo, dare mo demasen desita yo.> Hisyo ga osoku ni korareta.In each exercise, the context is described in English and one Japanese sentence (produced by an imaginary conversation partner) is given for the learners to respond to.Japanese writing systems were not involved in the present study.5 The correct answer to the question in FigureII is something like Ee, hisyo ni osoku koraremasita. Suppose a student typed an incorrect sentence at the bottom of the screen (Figure II), such as Hisyo ga osoku ni korareta, which contains several errors. Figure III illustrates the process of analyzing this incorrect sentence. First, the sentence is sent to the NLP analyzer. The NLP analyzer generates the output messages which provide grammatical information about the sentence. We see here one error message (i.e., particle error) in the output messages for this learner's response. The NLP analyzer contains a set of syntactic rules which can parse not only grammatical sentences butalso ungrammatical ones (e.g., missing or wrong use of particles, wrong verbal conjugation, etc.). If an ungrammatical sentence is parsed, error messages will be generated. The output messages are sent to the pattern matching program to check whether the learner's response is appropriatefor the question in the exercise. For example, this exercise has a list of grammatical information such as: the required words are ee, hisyo, ni, and osoku; the agent is hisyo; the citation form of the verbal predicate is kuru; the verbal predicate should be affirmative, perfective, distal, and passive, and so forth. The pattern matching program checks whether this grammatical information matches the output messages about the learner's response sent from the NLP analyzer. This program finds that ee is missing, hisyo is "subject" in the learner's response but should be "agent" in this question, and the verbal predicate in the learner's response is in the direct style but should be in the distal style. In short, several errors are detected by the NLP analyzer and the pattern matching program. The learner's errors are classified into vocabularyerror, particle error, verbal predicate error, and nominal modifier error. The final feedback is shown on the screen as seen in Figure IV.Figure III : Analysis of the Response (Incorrect Response)Learner’s responseHisyo ga osoku ni koraretaNLP analyzerFeedbackIn this study, the error analysis was focused on the learners' "intralingual errors" (13), which originate within the structure of the target language, such as overgeneralization, ignorance of rule restrictions, incomplete application of rules, and false concepts hypothesized (e.g., wrong particles, wrong verbal forms, styles, or aspects, incomplete conjugation, wrong vocabulary, etc.). The feedback messages for particle errors were developed based on the "psycholinguistic processing approach" (2), which focuses on the learners' problems derived from unclear conceptualization of the target language and instructs the learners in ways to understand different concepts, functions, and semantic relations existing in the target language. The system allows language instructors to create feedback messages themselves, depending on their instructional focus.Figure V summarizes the difference between conventional CALI and intelligent CALI systems. Conventional CALI systems do not analyze the learner's response on the basis of a linguistic analysis of the target language. That is, conventional CALI systems do not have an NLP analyzer. They simply compare the response with a machine-stored correct or incorrect version by a simple pattern matching technique, and canned feedback is presented if the learner's response matches an anticipated error.Figure IV : Feedback in the Nihongo-CALI ExerciseOptions1. Respond to the following conversation, according to the situation described.(1) Your colleague told you that he called your office this morning but nobody answered. Respond that yes, the secretary came in late (i.e., you were affected by the secretary's coming in late).Colleague: Kesa zimusyo ni denwa simasita kedo ,dare mo demasen desita yo.Your response: Hisyo ga osoku ni korareta.Read the following messages:<Vocabulary error>EE or HAI is missing. (trivial error)<Particle error>GA is not expected to be used here.In your sentence, HISYO is the 'subject' of the passive (the one that is affected by the action), but it should be the 'agent' of the passive (the one who performs the action and affects the subject). Use the particle NI to mark it.OSOKU modifies the verbal predicate without the particle NI.<Verbal predicate error>Since you are talking with a colleague, use the distal-style (-masu form).Figure V: Conventional CALI and Intelligent CALI SystemsLearner’s responseConventional CALINLP analyzerOutput messagesPattern MatchingPark and Seidel point out that intelligent computer-assisted instruction (ICAI) is fundamentally different from traditional computer-assisted instruction (CAI) or computer-based instruction (CBI) in terms of the basic philosophies underlying the structures and development processes of the systems. For example, the early forms of CBI were strongly influenced by Skinnerian behaviorism (which focuses on stimulus-response links and reinforcement along with predetermined learning sequences). A real expectation of Skinner's frame-based, programmed instruction is to function as an effective reinforcer for students in the "preparation" stage, which involves simple knowledge and skills. On the other hand, many ICAI projects have sought to better understand learners' cognitive structures and processes involved in learning the given task and diagnostic processes of instruction. Bonner (p. 31) describes the computer courseware continuum from "frame-based CAI" to "intelligent tutors": "Frame-based CAI is a sound choice for teaching simpler knowledge and skills early in a curriculum when students have a minimum of knowledge about a subject .... This is a situation where the instructional designer can anticipate enough of the students' responses to provide minimal limitations in interacting with the knowledge. Intelligent tutors are a good choice for more complex kinds of learning, when the student knows enough about a subject to work with it in a problem solving context .... The more complex the task, the more difficult it is to specify the responses required to keep a frame-based approach individualized enough." The point here is that we need to use both kinds of CAI (frame-based CAI and intelligent tutors), depending on the purpose and the type of learning task, rather than replacing one with another. Accordingly, the present study hypothesized that traditional CALI systems can be suitable for vocabulary and conjugation exercises which focus on low level (word-level) processing and automatization of simple knowledge and skills. This focus is what Skinner calls the preparation stage, needed before moving to the more productive and creative stage. On the other hand, intelligent CALI systems can be more suitable for exercises of sentence production and case particles which involve high level (sentence-level) processing and complex knowledge of grammar and concepts. Teaching such skills requires more detailed explanations of the learners' errors.One of the purposes of the present study was to provide empirical data to establish what kind of error analysis and feedback intelligent CALI should provide. A more specific question was what kind of error analysis and feedback could be effective for the adult learners of Japanese as a foreign language to develop their grammatical skills and language accuracy. The study compared two versions of the Nihongo-CALI exercises, the traditional CALI (T-CALI) exercises (involving conventional feedback) and the intelligent CALI (I-CALI) exercises (providing sophisticated feedback), and investigated whether the differences in the amount and quality of feedback, based on error analysis, would affect the learners' performance on tests of production of Japanese passive sentences. The empirical data obtained by Nihongo-CALI would tell us what difficulties the learners had in the process of learning Japanese and what feedback was helpful to learners for understanding their mistakes and correcting them. The study also sought to determine whether significant differences exist in the learners' attitudes toward T-CALI andI-CALI exercises.The procedures of the present study involved two steps: 1) the development of the two different types of exercises; and 2) the evaluation of these programs. The next section describes these steps.METHODOLOGYThe first stage of the study was to develop two different types of exercises (T-CALI and I-CALI exercises) for learning and practicing Japanese. The Nihongo-CALI system described in the previous section was thus implemented in two versions: T-CALI and I-CALI. That is to say, both T-CALI and I-CALI used the NLP analyzer to process and analyze the learner's input, but provided only limited feedback in simulating the traditional CALI approach. We thus eliminated artifactual differences stemming from differences in computer processing speed and range of coverage of errors. Our focus was specifically on differences in the amount and quality of feedback associated with the T-CALI and the I-CALI exercises. Although traditional CALI varies in the types of error diagnosis and feedback used, the study followed the most sophisticated type of traditional CALI for error diagnosis and feedback such as that used in the PLATO system (3). That is, the traditional CALI exercises in the present study provided feedback about what was wrong (i.e., what word was missing, incomplete, or not expected to be used), while the intelligent CALI exercises provided not only the above information but also detailed explanations of why responses were wrong. These two types of exercises aimed to develop the students' grammatical competence. They are based on a grammatical syllabus around which functions, situations, and communicative activities (responding to questions asked by an imaginary conversational partner) are grouped. They used the same grammatical structures, the same vocabulary items, the same question content, and the same hypertext environments (where the students had free access to grammar notes, vocabulary lists, lexical lookup, and the exercises). Spoken interaction was not included in this program. The following example shows the type of feedback provided by T-CALI and I-CALI. A learner was presented the task, "At a party your friend has asked if there are still Japanese drinks (left). Respond that no, the students went and drank all the Japanese beer (i.e., you were affected by the students' drinking all the Japanese beer)." A Japanese sentence, Nihon no nomimono ga mada aru? was provided to respond to. The correct response is Uun, gakusee ni nihon no biiru o zenbu nomareta yo. Suppose the learner typed, Uun, gakusee ga nihon no biiru o zenbu nomaremasu, "No, the students will be affected by someone's drinking all the Japanese beer." For this response, T-CALI provided this feedback: "GA is not expected to be used here. NI is missing. NOMAREMASU is wrong." I-CALI, however, provided not only these messages but also more detailed grammatical explanations about the errors, e.g., "In your sentence, GAKUSEE is the 'subject' of the passive (the one that is affected by the action), but it should be the 'agent' of the passive (the one who performs the action and affects the subject). Use the particle NI to mark it. The predicate you typed is in the imperfective form. Change it to perfective. Since you are talking with your friend and your friend is using the direct-style (casual style), use the direct-style for your response." In short, I-CALI provided more information on particle errors (i.e., errors in the use of wa, ga, o, ni, e, de, no, kara, and made) and verbal predicate errors (i.e., errors in style, tense, negative or affirmative, passive or non-passive, conjugation of passive forms, and gerund + iru or non-gerund). When the students did not come up with a right answer on the third try, the correct answer could be selected from an option menu.In the next stage, an experiment was performed to evaluate the comparative effectiveness of these two types of Nihongo-CALI exercises. Thirty-four subjects in the second-year Japanese language course at the University of Pittsburgh participated in the experiment. Eachsubject had six one-hour sessions (two classroom sessions and four computer sessions) over the course of a week. The first session was conducted in the classroom: the subjects were given brief explanations about passive structures and a written test to assess their knowledge of basic grammar. Based on the results of the assessment test, the subjects were paired and randomly assigned to either the I-CALI or T-CALI group. From the second to fifth sessions (the computer sessions), the students learned and practiced the passive structures through the Nihongo-CALI exercises (ninety exercises in all).6 At the beginning of each computer session, the students spent about seven minutes reading the grammar notes (which explained the Japanese passive construction) in the Nihongo-CALI program, and then went to the exercises. Since there are three types of passive in Japanese, i.e., the direct, indirect (adversative), and honorific passive, the first session practiced the direct passive, the second the adversative passive, the third the honorific passive, and the fourth session was used to review all types of passive. While doing the exercises, the students could refer to the vocabulary hints, access the lexical lookup (on-line dictionary), or go back to the grammar notes (on-line grammar pages). The students were not aware that different types of feedback were being compared. No additional instruction was provided during the experimental sessions. In the last computer session, the subjects were asked to answer a questionnaire. The sixth session was conducted in the classroom: the subjects took an achievement test. As a follow-up, in the week following the experimental sessions, the students had speaking practice using the passive sentences in the regular recitation class for thirty minutes. Three weeks after the experiment, the students took a retention test.RESULTSTwenty questions were given on the achievement test; a perfect score was thirty-four. The achievement test followed the same format and content used in the Nihongo-CALI exercises. Table I shows the means and standard deviations of the I-CALI and T-CALI groups for the achievement scores, and the result of the t-test for these correlated (matched) groups.Table I: Results of the Achievement Test (Scores Out of 34)__________________________________________________________________Mean SD t sig of t__________________________________________________________________3.3I-CALI 27.9_____________________________________ 2.18 .053.8T-CALI 26.9__________________________________________________________________The t score value shows that the difference between I-CALI and T-CALI groups in the learners' achievement on the target structures is statistically significant at the .05 level. Table II presents the total number of errors and the number of each type of error found on the achievement test. The numbers of errors on the achievement test show that the greatest difference between the two groups resulted from particle errors (i.e., the T-CALI group made forty-four more particle errors than the I-CALI group).。