Collaborative Dialogue with a Learning Companion as a Source of Information on Student Reas

Collaborative Dialogue with a Learning Companion as aSource of Information on Student ReasoningEva L. RagnemalmDept. of Computer and Information Science, Linköping University, Sweden.E–mail: elu@ida.liu.seAbstract: This report focuses on a problem within the area of Intelligent Tutor-ing Systems; that of analysing student’s reasoning (student diagnosis). A novel approach to collecting information for this analysis, complementary to tradi-tional student modelling techniques, is presented. This technique is based on using a Learning Companion, a computer based agent, as a collaboration partner to the student. In the dialogue between the student and the Learning Companion,information on their problem-solving process is revealed. This information would then be extracted and used for student modelling purposes. Analysis of the proposed solution is commenced in a small experiment and an explorative implementation described here.Keywords: Learning Companion Systems, Student Modelling, Student Diagno-sis, Collaborative Dialogue, Troubleshooting.1IntroductionOne of the important components in an Intelligent Tutoring System, ITS, is a Student Model . This is a collection of data about the current student which is used at runtime by the other components in the system in a number of different tasks. Such tasks include for instance planning the sequence of instruction, remediating misconceptions,generating feedback and explaining the reason for an error made by the student (see e.g. [23], [18], [12]). The complexity of the student modelling problem has caused researchers to question the necessity of student modelling (see e.g. [16] and for coun-ter–arguments [15]). This question will not be discussed further in this paper, in the following it is assumed that student modelling is worthwhile.In this paper student modelling is defined as the process of dynamically gathering information about the student and storing that in a student model. Student modelling is a part of Student Diagnosis , which is the process of analysing the student’s actions and converting that to a foundation for educational decisions [20].Most ITS adhere to some variation of the pedagogic principle of learning-by-doing.That is, most ITS are designed to allow the student to exercise the skill to be learned (e.g. LISP tutor [6], Sherlock [13], Sophie, [1], Scent [11]). It is in this context that stu-dent modelling takes place. The student modelling process is usually a covert activity,even though overt and negotiated student modelling have been proposed (e.g. [10],[16]). In covert student modelling student diagnosis is based on the input available as Presented at ITS’96, Montreal, 12-14 june 1996Pages 650-658 in the proceedings, Springer V erlagthe student solves the exercises presented by the ITS. The benefit of covert modelling is that the student is not distracted from the problem solving by (to him) irrelevant questions.The problem of student diagnosis can be subdivided into three phases [20]. These phases are the data acquisition phase, the transformation phase and the evaluation phase. Data acquisition refers to the collection and refinement of the input, including the abstraction of actual input (such as mouse movements or whatever peripherals are used) to concepts in the domain (such as an action or an answer). Transformation refers to the conversion done either by reconstructing the knowledge causing the stu-dent action or by applying knowledge of how to solve the exercise and comparing the result to the student’s action. The evaluation refers to the judgement of whether what the student is believed to know is correct or not in relation to the goal of the teaching situation. The student model is constructed either before or after the evaluation phase, thus student modelling covers either the first two or all three phases.One problem in the data acquisition for covert student modelling is that of gather-ing sufficient data. If a procedure is taught, procedure oriented evaluation should be used, not goal oriented, but if the procedure is mental it will be inaccessible to the ITS.Techniques for extrapolating the student’s unobservable behaviour from what can be observed include plan recognition and path finding techniques (see e.g. [23]). These techniques have the drawback that they are dependent on the complexity of the exer-cise the student is doing. When there are many paths or plans that explain observed activities, they quickly become intractable.There are also techniques that attack the problem of data acquisition by having the student provide additional information to the system, information which is normally not expressed during problem solving. FITS [18] and EPIC [22] require that the stu-dent specify what subproblem he wishes to work on. Other systems provide a tool interface through which information is collected while the student uses the tool in the problem solving. An example of this is the program editor in the LISP tutor [6].This paper proposes another approach to this problem: the use of collaborative dia-logue with a Learning Companion, LC. Collaborative dialogue reveals the reasoning of the collaborating partners. The dialogue can thus provide additional information for the data acquisition process. First an example domain is presented. The alternate method is then presented and analysed. A preliminary experiment and explorative implementation are presented. These initial studies indicate that the method will prob-ably suffice but is computationally complex.2Example: student modelling of DiagnosisAn example of a situation where the data available is scarce is learning to diagnose faults in a pulp mill by practising with a simulator–based ITS. Diagnosis is here defined as the identification and remediation of faults in a process. The strategy of diagnosis is a cognitive procedure with 10 steps1 to be executed in specific order [21]. The steps are presented in Table1 and the observed sequence illustrated in Figure 1.1.The original list [21] contained 8 steps, (1-7, 10), but the author found evidence for another,the ordering of repairs, 8. The application of repair is also shown as an explicit step, 9.Table 1: The steps to take for a correct diagnosis.experts is significant. Conversely, the (mental) execution of the top–level strategy (which steps and the order they are taken) is thus significant indication of the student’s level of expertise.Expert strategy Novice strategyFig. 1.Models of the diagnostic strategy, employed by experts and novices respectively.Consider an ITS with the goal of teaching the novices to use the same strategy as the experts. The differences in the models imply that modelling the student’s strategy of diagnosis is necessary to judge his competence. The student’s strategy of diagnosis is thus an important part of the student model in this ITS. This strategy, however, ismental. The only step which is normally overt is the application of a repair, which shows as input to the simulator. This makes covert data acquisition difficult.The data acquisition techniques mentioned previously are not suited to this prob-lem. Path finding and plan recognition are unsuitable since there are many possible reasons for applying one specific repair. The problem is of exponential complexity.It would be possible to have the student specify to the ITS which subproblem (i.e. strategy step) he is working on. The drawback with this is that in order to be non–obtrusive, a menu would be the logical choice for interaction. This would provide the student with reminders of the steps that are often forgotten, which would not be present in the real working situation. Similarly, a tool on top of the simulator changes the task into something different from the real working situation. These solutions thus seem to be less than ideal.3Collaborative dialogue as a source of informationA technique that is sometimes used by teachers is that of listening to two students talk-ing about a problem. A similar technique is used by researchers in decision making when studying the cognitive procedures of taking a decision. The task is converted into one requiring collaboration between two subjects and the dialogue protocol used as a source of data on the decision making [25]. Such collaborative dialogue could be tapped by a system capable of parsing it, and used in the student diagnosis.This approach, however, requires the presence of two students. One option is to use a computer–based agent as the collaboration partner. Chan and Baskin [4] propose the concept of Learning Companion Systems, LCS, which are ITSs equipped with an additional agent, the Learning Companion, LC. The LC is a pedagogic tool in the form of a “fellow student” that can either collaborate, teach or compete with the student. LCs have been constructed for competition and reciprocal tutoring [3], [5], concurrent collaboration [8], [7], and learning–by–teaching [17].The purpose of the LC as proposed here is the possibility of using the dialogue aris-ing from the collaboration as a source of data for the student modelling process. In this case we are only interested in the collaborative LC, since the interesting dialogue does not occur in the competitive or teaching–related settings.The situation envisioned is a LCS, where the task of the simulator–based ITS com-ponent is to present exercises to the student, based on its student model, and evaluate the results. The task of the LC is to collaborate with the student on solving the exer-cises. The LC’s reasoning should be restricted so that the LC does not rush off and solve the problem on its own, since the student should be working on the problem as well. The ITS then takes the arising dialogue and uses it as input to the student model-ling procedure. This input will reflect the student’s reasoning in greater detail than the input provided to the simulator alone.Given the greater detail of student input, existing student modelling techniques would then be used in the ITS. For instance, issue tracing [2] can be used, although the definition of recognisers will rely on the quality of the speech recognition system in the case where spoken language is used.4Design of a Learning CompanionCreating a LC is a challenging problem. In general the LC is a kind of computer–based intelligent agent and as such appears to be AI complete. The observation that collabo-rative dialogue reveals reasoning was based on spoken natural language dialogue, thus the LC may need to communicate using spoken natural language. In order to collabo-rate, it needs to be capable of solving the problem the student is presented with. To be perceived as a believable fellow student, it might also have to learn along with the stu-dent. These are complex problems.It is possible, however, that the specific LC required to cause the student to reveal his reasoning need not be so complex. It must not be so simple as to cause the student to be irritated with it, but it might not require complete capacity on all the above aspects. Its main purpose is to encourage the student to speak his mind, and Weizen-baum’s Eliza [24] and Mauldin’s Julia [9] show the extent to which a simple design can be acceptable.A preliminary study of the feasibility of using a LC to elicit collaborative dialogue which could be used to inform the student modelling process was performed. It focused on three questions:1.Does the collaborative dialogue really reflect such a high level strategy as that ofdiagnosis? The original models captured by Schaafstal [21] were based on think–aloud protocols and deep interviews. Is it possible to get enough information from uncontrolled dialogue? Is the information that is needed to identify the student’s strategy really present in dialogue generated by collaborative problem solving? 2.How computationally complex is the problem of extracting the pertinent informa-tion from the spoken dialogue? Specifically, are keywords usable (making the speech–recognition technique keyword–spotting [14] usable)?3.How complex would the construction of this simplified “intelligent” agent be?How would it be designed?A small experiment was carried out in order to study the first question, and the data was also used in a preliminary study of the second. The third question was studied in an explorative implementation of a prototype LC. The experiment and the implementa-tion are briefly described below. A more detailed description can be found in [19].5Dialogue experimentIn order to examine the information content of collaborative dialogue, an experiment on diagnosis of an evaporator in a collaborative setting was performed. A simulator was used to illustrate the fault situation, and the subjects were to identify the fault and compensate for it. Five subjects with no previous skill in process control or diagnosis were used. They were instructed in the expert strategy and told to collaborate on the exercises.The subjects were divided into three pairs2 of subjects, for each pair two exercises were video-taped and transcribed, divided into utterances based on speaker and con-tent. Table2 shows a sample of dialogue at the beginning of an exercise.The state-2.One subject participated twice, in order to see if a learning effect was achieved during theexperiment. None was detected.ments were analysed and classified by two different raters as to what diagnostic step was reflected, if any.All the ten steps could be found in the dialogue. The dialogue Table 2: Collaborative dialogue, diagnosing an evaporatorcontained a high percentage of statements classified as coming from one of the ten steps (68% classified, another 26% difficult to classify, as evidenced by different clas-sification by the raters, but still classifiable).The fact that all the ten steps could be identified indicates that sufficient informa-tion to recognise the different steps was present in the dialogue. Provided that suffi-cient techniques for extracting that information can be developed, it should be possible to use the dialogue for student modelling. The high percentage of classifiable state-ments indicates that much of the dialogue is task oriented, eliminating the need to allow for small talk.With respect to the second question, an attempt at finding keywords for identifying strategy steps failed. For instance, sets of keywords that cover more than 50% of the utterances pertaining to Identification of Symptoms start yielding false hits. This implies that the problem of information extraction requires further study.6Explorative implementation: ElseAs a first step of assessing the possibility of simplifying the design of the LC, an explorative implementation of key aspects of an LC was made. The program was called Else. The dialogue from the experiment above was studied in order to identifyrecurrent characteristics and the behaviour of the subjects was used as a source of inspiration on the behaviour of Else. The existing LCs were thought to be too different from the aims of the current one to be copied. Only the core functionality (problem solving and simple dialogue management) were included in Else, no parsing or lan-guage generation was implemented. Else was designed to be capable of solving a diag-nostic problem in the simulator and to interact with the student in the process. A more complete report on this implementation is found in [19].The first of the identified properties was the fact that the majority of the utterances consisted of straightforward statements of facts (claims, cf. statements 4–10 in Table2) and a minor part was a claim backed by one or more observations (as in state-ments 15–17 in Table2). These two types of claims are formed in terms of results of subproblems in the strategy. Discussions related to the subproblems themselves (as in statements 50-51 in Table2) account for no more than 4% of all statements, and were thus thought less important at this point. This led to a decision to leave out models of the subproblems and the evaporator functionality from the design, making Else incapa-ble of discussing anything but the top level strategy. This decision is also supported by the finding by Schaafstal [21], that the novices were already skilled in the subprob-lems, thus not necessarily requiring training of them.Another property observed was that parts of the dialogue were not dialogue but intertwined monologues, as in statements 2–10 in Table2. In these parts it was not possible to identify any underlying plan or intention except for the diagnostic strategy itself and subsequently no other overall plan was identified either. Else has a model of diagnosis but no explicit plan for the dialogue or the collaboration and contains no model of the collaboration partner (the student). It is strictly reactive.The fact that much of the interaction consisted of providing an alternate result or the next result led to the view that collaboration could be seen as using the compan-ion’s expressed results in the private problem solving, and expressing subsequent results. This was realised through an architecture based on the separation of problem solving and dialogue management. Although they interact, neither component controls the other. The fact that these components together cause Else to appear collaborative, since Else uses the student’s utterances in its own reasoning, can be seen as an emer-gent property. Else was also given the ability to choose whether to attend the student’s statements or not, based on a collaboration mood.The subjects were also found to repeat themselves and to express themselves in non–grammatical and incomplete sentences. The first behaviour was allowed for in the implementation by having Else “forget” facts, while the second will have to be imple-mented in the parsing part of a complete LC. The interaction with Else is purely sym-bolic, so no parsing is done. Else allows for interruptions and also hesitating students by giving the option to “speak” or be silent regardless of whether the student is cur-rently speaking.Else is capable of participating in a dialogue similar to that in statements 4–17 in Table2, but can not produce or respond to statements like those in 50–51. Else is an explorative implementation, and further analysis of its properties is required. Specifi-cally, the sufficiency of collaboration as an emergent property requires further study. The basic criteria is that Else must be capable of eliciting dialogue from the student,such that the student’s strategy is revealed. The data acquisition in student modelling also remains to be designed for extracting the novel data provided in the dialogue.7SummaryThis paper introduced the problem of how to trace a student’s reasoning for student modelling purposes, specifically when modelling the student’s strategy of diagnosis in a simulator–based environment.The proposed solution to the problem of data acquisition is based on the use of col-laborative problem–solving dialogue. A small experiment supported the belief that the dialogue between two students collaborating on diagnosing a fault presented on a sim-ulator does reflect their reasoning, and thus reveals the diagnostic strategy they apply. The data thus elicited would then be used by the student modelling process to build and maintain a student model. The experiment also indicated that the problem of pars-ing the spoken language and constructing recognisers for the student modelling proc-ess is a complex problem still to be solved.The explorative implementation of a prototype LC, Else, indicated the possibility of modelling collaboration as an emergent phenomenon, where dialogue and problem solving are modelled as equal processes, neither controlling the other. Further research will continue the exploration of the design of a LC.References[1]Brown, J.S., Burton, R., and de Kleer, J. Pedagogical, natural language and knowledgeengineering techniques in SOPHIE I, II, and III. In Sleeman, D. and Brown, J.S., editors, Intelligent Tutoring Systems, chapter11, pages 227–282. Academic Press, 1982.[2]Burton, R.R. and Brown, J.S. An investigation of computer coaching for informal learningactivities. In Sleeman, D. and Brown, J.S., editors,Intelligent Tutoring Systems, chapter4, pages 79–98. Academic Press, 1982.[3]Chan, T.-W. Integration kid: A Learning Companion system. In Proceedings of IJCAI’91,pages 1094–1099, Sydney, Australia, 1991.[4]Chan, T.-W. and Baskin, A.B. Learning companion systems. In Frasson, C. and Gauthier,G., editors,Intelligent Tutoring Systems: At the Crossroads of Artificial Intelligence andEducation, pages 6–33. Ablex, New Jersey, 1990.[5]Chan, T.-W. and Chou, C.-Y. Simulating a learning companion in reciprocal tutoring sys-tems. In Proceedings of Computer Supported Collaborative Learning (CSCL’95), on-line version, 1995. /cscl95/chan.html (accessed Nov 7, 1995). [6]Corbett, A.T. and Anderson, J.R. LISP Intelligent Tutoring System: research in skillacquisition. In Larkin, J.H. and Chabay, R.W., editors,Computer Assisted Instruction and Intelligent Tutoring Systems: Shared Goals and Complementary Approaches, chapter3, pages 73–109. Lawrence Erlbaum Associates, 1992.[7]Dillenbourg, P. and Self, J.A. PEOPLE POWER: A human-computer collaborative learn-ing system. In Frasson, C., Gauthier, G., and McCalla, G.I., editors,Proceedings of ITS’92, pages 651–660. Springer V erlag, 1992.[8]Dillenbourg, P.J.Human-Computer Collaborative Learning. PhD thesis, University ofLancaster, Department of Computing, 1991.[9]Foner, L.N. What’s an agent anyway? a sociolgical case study. Technical Report 93-01,Agents Group, MIT Media Lab, MIT Media Laboratory, 20 Ames St. Camebridge, MA02139., 1993. Available via Foner’s home page /people/foner/ (as of 19th of March 1996).[10]Goodyear, P. and Tait, K. Learning with computer-based simulations: Tutoring and studentmodelling requirements for an intelligent learning advisor. In Learning and instruction: European research in an international context, pages 463–481, 1991. Proceedings of the 3rd conference of the European Assoc. for Research on Learning and Instruction, 1989. [11]Greer, J., Mark, M., and McCalla, G. Incorporating granularity-based recognition intoSCENT. In Bierman, D., Breuker, J., and Sandberg, J., editors,Proc. of the 4th Interna-tional Conference on AI and Education, pages 107–115, Amsterdam, 1989.[12]Holt, P., Dubs, S., Jones, M., and Greer, J. The state of student modelling. In Greer, J.E.and McCalla, G.I., editors,Student Modelling: The Key to Individualized Knowledge-Based Instruction, NATO ASI Series F, pages 3–35. Springer-Verlag, 1994.[13]Lesgold, A., Lajoie, S., Bunzo, M., and Eggan, G. Sherlock: A coached practice environ-ment for an electronics troubleshooting job. In Larkin, J.H. and Chabay, R.W., editors, Computer-Assisted Instruction and Intelligent Tutoring Systems: Shared Goals and Com-plementary Approaches, chapter7, pages 201–238. Lawrence Erlbaum Associates, 1992.[14]Markowitz, J. Keyword spotting in speech.AI Expert, pages 21–25, October 1994.[15]McCalla, G. The central importance of student modelling to intelligent tutoring. In Costa,E., editor,New Directions for Intelligent Tutoring Systems, pages 107–131. Springer-Ver-lag, 1992.[16]Newman, D. Is a student model necessary? Apprenticeship as a model for ITS. In Bierman,D., Breuker, J., and Sandberg, J., editors,Proc. of the 4th International Conference on AIand Education, pages 177–184, Amsterdam, 1989.[17]Nichols, D. Issues in designing learning by teaching systems. AAI/AI-ED 107, ComputingDepartment, Lancaster University, Lancaster, LA1 4YR, UK, July 1994. Accessed via Internet, Sept. 1995. /computing/research/aai-aied/.[18]Nwana, H.S. User modeling and user adapted interaction in an intelligent tutoring system.User Modeling and User-Adapted Interaction, (1):1–32, 1991.[19]Ragnemalm, E.L. Towards student modelling through collaborative dialogue with a learn-ing companion. Licentiate thesis Thesis No 482, Linköping University, 1995.[20]Ragnemalm, E.L. Student diagnosis in practice; Bridging a er Modelling and UserAdapted Interaction, 5(2):93–116, 1996.[21]Schaafstal, A.M.Diagnostic Skill in Process Operation: A Comparison Between Expertsand Novices. PhD thesis, University of Groningen, Institute for Perception TNO, P.O. Box 23, 3769 Soesterberg, The Netherlands, September 1991.[22]Twidale, M. Intermediate representations for student error diagnosis and support. In Bier-man, D., Breuker, J., and Sandberg, J., editors,Proc. of the 4th International conference on AI and Education, pages 298–306, Amsterdam, May 1989.[23]VanLehn, K. Student modeling. In Polson, M.C. and Richardson, J.J., editors,Founda-tions of Intelligent Tutoring Systems, chapter3, pages 55–78. Lawrence Erlbaum Associ-ates, 1988.[24]Weizenbaum, J. ELIZA – a computer program for the study of natural language communi-cation between man and munications of the ACM, 9(1):36–45, 1966. [25]Woods, D.D. Process–tracing methods for the study of cognition outside of the experimen-tal laboratory. In Klein, G.A., Orasann, J., Calderwood, R., and Zambok, C., editors,Deci-sion-Making in Action: Models and methods, chapter13, pages 228–251. Ablex publishing, 1993.。