Spoken Dialogue Management Using Hierarchical Reinforcement Learning and Dialogue Simulatio

Spoken Dialogue Management Using Hierarchical Reinforcement Learning and Dialogue SimulationHeriberto Cuay´a huitlSupervisors:Prof.Steve Renals and Dr.Oliver LemonPhD Research ProposalInstitute for Communicating and Collaborative SystemsSchool of InformaticsUniversity of EdinburghOctober2005AbstractSpeech-based human-computer interaction faces several difficult challenges in order to be more widely accepted.One of the challenges in spoken dialogue management is to control the dialogueflow(dialogue strategy)in an efficient and natural way.Dialogue strategies designed by humans are prone to errors,labour-intensive and non-portable,making automatic design an attractive alternative.Previous work proposed addressing the dialogue strategy design as an op-timization problem using the reinforcement learning framework.However,the size of the state space grows exponentially according to the state variables taken into account,making the task of learning dialogue strategies for large-scale spoken dialogue systems difficult.In addition, learning dialogue strategies from real users is a very expensive and time-consuming process, making automatic learning an attractive alternative.To address these research problems three lines of investigation are proposed.Firstly,to investigate a method to simulate task-oriented human-computer dialogues at the intention level in order to design the dialogue strategy au-tomatically.Secondly,to investigate a metric to evaluate the realism of simulated dialogues. Thirdly,to make a comparative study between hierarchical reinforcement learning methods and reinforcement learning with function approximation,in order tofind an effective and efficient method to learn optimal dialogue strategies in large state spaces.Finally,a timeline for the completion of this research is proposed.Keywords:Spoken dialogue systems,probabilistic dialogue management,human-computer dialogue simulation,user modelling,hidden Markov models,dialogue optimization,dialogue strategies,Markov decision processes,Semi-Markov decision processes,reinforcement learn-ing,hierarchical reinforcement learning,function approximation,dialogue systems evaluation.iiiAcknowledgementsThis research is being sponsored mainly by PROMEP(“PROgrama de MEjoramiento del Pro-fesorado”),part of the Mexican Ministry of Education(http://promep.sep.gob.mx).It is also being sponsored by the Autonomous University of Tlaxcala(www.uatx.mx).ivTable of Contents1Introduction11.1Spoken Dialogue Systems (1)1.2Motivation (2)1.3Proposal (3)1.4Research Questions (4)1.5Contributions (4)2Previous Work52.1Spoken Dialogue Management (5)2.2Human-Computer Dialogue Simulation (8)2.3Reinforcement Learning for Dialogue Management (10)2.4Spoken Dialogue Systems Evaluation (11)3Human-Computer Dialogue Simulation Using Hidden Markov Models133.1Introduction (13)3.2Probabilistic Dialogue Simulation (14)3.2.1The System Model (14)3.2.2The User Model (15)3.2.3The Simulation Algorithm (15)3.3Dialogue Similarity (16)3.4Experimental Design (17)3.4.1Training the System and User Model (17)3.4.2Evaluation Metrics (19)3.4.3Experiments and Results (19)3.5Conclusions and Future Directions (21)3.6Proposed Future Work (22)4Spoken Dialogue Management Using Hierarchical Reinforcement Learning254.1Introduction (25)4.2The Reinforcement Learning Framework (26)4.2.1Markov Decision Processes (26)4.2.2Semi-Markov Decision Processes (28)4.2.3Reinforcement Learning Methods (28)4.3Hierarchical Reinforcement Learning Methods (29)4.3.1The Options Framework (29)4.3.2The MAXQ Method (31)4.3.3Hierarchies of Abstract Machines (33)4.3.4Comparison of Methods (34)4.4Reinforcement Learning with Function Approximation (35)v4.5Experimental Design (36)4.5.1The Agent-Environment (36)4.5.2Evaluation Metrics (38)4.5.3Experiments (38)4.6Proposed Future Work (38)5Future Plans395.1Timetable (39)References41viChapter1Introduction1.1Spoken Dialogue SystemsSpeech-based human-computer interaction faces several difficult challenges in order to be more widely accepted.An important justification for doing research on this topic is the fact that speech is the most efficient and natural form of communication for humans.Currently,human-computer interaction is mainly performed using the following devices:keyboard,mouse and screen.There are many different and attractive reasons for interacting with computers us-ing speech.For instance,in hands-free and eyes-free environments(such as walking or driv-ing a car),in computer applications where providing information is tedious(such as search-ing/consulting information or booking a service),in mobile environments,or simply to have fun(such as talking with toys).Computer programs supporting interaction with speech are called“Conversational Interfaces”,computer programs supporting different modalities(such as speech,pen,and touch among others)are called“Multimodal Conversational Interfaces”, computer programs supporting only speech are typically called“Spoken Dialogue Systems”. The main task of a spoken dialogue system is to recognize user intentions and to provide co-herent responses until user goals are achieved.Figure1.1illustrates the architecture of a basic spoken dialogue system.Briefly,when the user speaks,the“input”components recognize user intentions and provide them to the“di-alogue manager”(DM),which consults information from a database and provides an answer to the user through the“output”components.The conversation is basically driven by three levels of communication:speech,words and intentions.Typically,the user provides speech signals using either a microphone or telephone.The“input”components receive a speech signal and provide user intentions:the speech signal is given to the“Automatic Speech Recog-nition”(ASR)component,which looks for the words corresponding to the given speech signal and passes them on to the“natural language understanding”(NLU)component,which looks for the intentions corresponding to the given words.The main task of the dialogue manager is to control theflow of the conversation in an effective and natural way by providing the best system intentions given the current user intentions,information from the database and history of the conversation.The“output”components are the counterpart of the input components, receiving a set of system intentions and providing a speech signal to the user:the system in-tentions are given to the“natural language generation”(NLG)component,which generates a contextually appropriate response and provides the corresponding words to the speech syn-thesis”(TTS)component,which provides the corresponding speech signal to the user.In this way,human-computer conversations are compounded by user turns and system turns in afinite iterative loop until user goals are achieved.The process described above still is a challenge for science and engineering.None of the12Chapter1.IntroductionFigure1.1:Basic architecture of a spoken dialogue system. components described above is perfect,even for simple tasks.The ASR component may pro-vide an incorrect sequence of words due to noise in the channel,noise in the environment or out-of-vocabulary words.The NLU component may provide an incorrect sequence of in-tentions due to the fact that a word sequence may be incorrect or may have several different interpretations.The DM component has a very challenging task due to the fact that the input components may convey incorrect user intentions,so the DM must work under uncertainty. In order to overcome the misunderstandings of the input components and choose the best sys-tem intentions it must use all possible knowledge in the conversation.In addition,the NLG component may provide a word sequence that may be unclear for the user.Finally,the TTS component usually provides an unnatural speech signal that may distract the user’s attention.All components in a spoken dialogue system may be simplified for well defined and simple task-oriented applications.For instance,the ASR may have a small vocabulary,the NLU may only provide semantic representation based on the utterance semantic tags,the DM may have a predefined controlflow of the conversation,the NLG may have predefined answers,andfi-nally,pre-recorded prompts may be used instead of TTS.However,even in simple applications sophisticated components are required for successful conversations in real environments.For instance,robust ASR and DM may significantly improve the performance of spoken dialogue systems.This research is focused on dialogue management for large-scale spoken dialogue systems, where the user may have several different goals in a single conversation(e.g.,some goals in the travel domain are:book a multi-legflight,book a hotel and rent a car).1.2MotivationThe main motivation behind this research is the fact that the automatic design of spoken dia-logue managers remains problematic,even for simple applications.Dialogue design(control flow of the conversations)is typically hand-crafted by system developers,based on their intu-ition about proper dialogueflow.There are at least three motivations for automating dialogue design:a)it is a time-consuming process,b)the difficulty increases according to the dialogue complexity,and c)there may be some design issues that escape system developers.Automating dialogue design using the reinforcement learning framework,based on learning by interaction within an environment,is a current research topic.However,the size of the state space grows exponentially according to the variables taken into account,making the task of learning optimal dialogue strategies1in large state spaces difficult.1.3.Proposal3Another related problem is how to learn the dialogue strategy automatically.Spoken dia-logue managers are typically optimized and evaluated with dialogues collected from lengthy cycles of trials with human subjects.But training optimal dialogue strategies usually requires many dialogues to derive an optimal policy and learning from conversations with real users may be impractical.An alternative is to use simulated dialogues.For dialogue modelling, simulation at the intention level is the most convenient,since the effects of recognition and understanding errors can be modelled and the intricacies of natural language generation can be avoided(Young,2000).Simulated dialogues must incorporate all possible outcomes that may occur in real environments,including system and user errors.Previous work has addressed this topic by learning a user model in order to plug it into a spoken dialogue system.However,little attention has been given to methods for simulating both sides of the conversation(system and user).Furthermore,there has been a lack of research on evaluating the realism of simulated dialogues.Some potential uses of simulating the system and user are:a)to acquire knowl-edge from both entities,b)to learn optimal dialogue strategies in the absence of a real spoken dialogue system and real users,and c)to evaluate spoken dialogue systems in early stages of development.The primary aims of my research proposal are:1.To investigate a method for simulating task-oriented human-computer dialogues on bothsides of the conversation(system and users).This is important because small dialogue data sets may be expanded for purposes of optimization and evaluation,and because knowledge from both entities may be acquired in order to improve the interaction.2.To investigate a metric for evaluating the realism of simulated dialogues.This is impor-tant because different system/user models may be compared in a more reliable way,so that the best models may be used for simulation.3.To investigate an efficient reinforcement learning method for learning optimal dialoguestrategies in large state spaces.This is beneficial to optimize dialogue strategies for dialogue managers with many different variables,which is useful not only for spoken dialogue systems but also for multimodal conversational interfaces,where the number of variables increases.1.3ProposalI propose a two-stage research project:thefirst stage focused on human-computer dialogue simulation and the second stage focused on learning optimal dialogue strategies.In thefirst stage I propose to investigate a novel probabilistic method to simulate task-oriented human-computer conversations at the intention level using hidden Markov models with rich structures.In addition,for evaluating the simulation method I propose to investigate a novel metric to measure the realism of the simulated dialogues,based on a probabilistic ap-proach of dialogue similarity and potentially combined with other metrics previously proposed in the natural language processingfield that may help to correlate dialogue realism.For the second stage I propose to perform a comparative study of three hierarchical rein-forcement learning methods proposed in the machine learningfield.This proposal is motivated by the fact that hierarchical methods have proven to learn faster,with less training data and have not been applied to spoken dialogue systems with large state spaces.In addition,I propose to compare the hierarchical methods against a function approximation method,which is another alternative for addressing optimization in large state spaces.Thefirst stage will be used as a4Chapter1.Introduction valuable resource for learning dialogue strategies automatically,and the second stage will be a valuable resource for learning dialogue strategies for large-scale spoken dialogue systems.Finally and for both stages,I propose to perform experiments using the2001DARPA Communicator corpora,which is annotated with the DATE annotation scheme and consists of ∼1.2K dialogues in the domain of travel information(multi-legflight booking,hotel booking and rental car).If time and resources permit I will perform experiments using a real spoken dialogue system and real users within the same domain.1.4Research QuestionsIn this research I aim to provide answers to the following questions:1.How can a small corpus of dialogue data be expanded with more varied simulatedconversations?There are two challenges in this question:a)how to predict system behaviour,and b)how to predict user behaviour.Due to the fact that probabilistic models will be used for this purpose,incoherent dialogues may be generated.Therefore,exploitation of knowledge from different variables must be taken into account.Preliminary experiments suggest promising results,and answers to this question will be driven by the following assump-tion:the more knowledge they have,the more accurate they are.An important problem to face in this question is data sparsity due to the small amount of data used to train the models.2.How can the realism of simulated human-computer dialogues be evaluated?This question is difficult due to the fact that it is not known if simulated dialogues may occur in real environments.Nevertheless,the answers to this question will be driven by the following assumption:the more similar the simulated dialogues are to the real ones, the more realistic they are.Another possible direction is through evaluating the utility of simulated dialogues under the following assumption:the more useful(for optimization or evaluation)they are,the more accurate they are.3.How can optimal dialogue strategies be learnt for spoken dialogue systems withlarge state spaces?There are two researchfields to address for this question:a)hierarchical reinforcement learning,and b)reinforcement learning with function approximation.There are sev-eral methods for eachfield and potential methods will be selected in order to performa comparative study,which may reveal potential application of specific methods in spo-ken dialogue systems.Experiments in this topic must be designed carefully in order to evaluate performance,computational cost and portability to other domains.1.5ContributionsThis research intends to advance the current knowledge in spoken dialogue systems according to the following expected contributions:1.A method to generate human-computer simulated dialogues at the intention level.2.A metric to evaluate the realism of human-computer simulated dialogues.3.A comparative study of reinforcement learning methods to learn optimal dialogue strate-gies in large state spaces.Chapter2Previous WorkThis chapter summarizes previous work that supports the proposal described in this document. The topics investigated and related to this research are:spoken dialogue management,human-computer dialogue simulation,reinforcement learning for dialogue management and evaluation of spoken dialogue systems.A brief description of each topic is provided,with references to relevant related work.Finally,at the end of each section a list of research gaps is provided. 2.1Spoken Dialogue ManagementThe main task of a spoken dialogue manager is to control theflow of the conversation between the user and the system.More specifically,a dialogue manager in task-oriented applications must gather information from the user(e.g.,“Where do you want to go?”),possibly clarifying information explicitly(e.g.,“Did you say London?”)or implicitly(e.g.“Aflight to Lon-don.For what date?”),resolve ambiguities that arise due to recognition errors(e.g.,“Did you say Boston or London?”)or incomplete specifications(e.g.,“On what day would you like to travel?”).In addition,the dialogue manager must guide the user by suggesting subsequent sub-goals(e.g.,“Would you like me to summarize your trip?”),offer assistance upon request(e.g.,“Try asking forflights between two major cities.”)or clarification,provide alternatives when the information is not available(e.g.,“I couldn’tfind anyflights on United.I have2Alaska Airlinesflights...”),provide additional constraints(e.g.,“I found tenflights,do you have a pre-ferred airline?”),and control the degree of initiative such as system-initiative(“What city are you leaving from?”)or mixed-initiative(e.g.,“How may I help you?”).The dialogue manager can influence other system components in order to make dynamic adjustments in the system such as:vocabulary,language models or grammars.In general,the goal of a spoken dialogue manager is to take an active role in directing the conversation towards a successful,effective and natural experience.However,there is a trade-off between increasing userflexibility and increasing system understanding accuracy(Zue&Glass,2000).There are several architectures for designing spoken dialogue managers,which can be broadly classified as follows:•State-Based:In this architecture the dialogue structure is represented in the form ofa network,where every node represents a question,and the transitions between nodesrepresent all the possible dialogues(McTear,1998).•Frame-Based:This architecture is based on frames(forms)that have to befilled by the user,where each frame contains slots that guide the user through the dialogue.Here the user is free to take the initiative in the dialogue(Goddeau et al.,1996)(Chu-Carroll, 1999)(Pieraccini et al.,2001).56Chapter2.Previous Work •Agenda-Based:This architecture is based on the frame-based,but builds a more com-plex data structure(using dynamic trees)for conversations in more complex domains (Rudnicky&Xu.,1999)(Wei&Rudnicky,2000)(Bohus&Rudnicky,2003).•Agent-Based:(Allen et al.,2001a)(Allen et al.,2001b)propose an architecture where the components of a dialogue system are divided into three areas of functionality:in-terpretation,generation and behaviour.Each area consists of a general control module and they communicate with each other,sharing information and messages.The agent responsible for controlling theflow of the conversation is mainly the behavioural agent.Another approach using collaborative agents is COLLAGEN,which mimics the relation-ships between two humans(agents)collaborating on a task involving a shared artifact.The collaborative manager supports mixed-initiative by interpreting dialogue acts of the tasks and goals of the agents(Rich&Sidner,1998).•Information State Approach:This architecture is based on the notions of information state,which represents information to distinguish dialogues,to represent previous actions and to motivate future actions(Larsson&Traum,2000).An important dialogue data collection from several different systems using different fea-tures is the DARPA Communicator corpora(available through LDC)(Walker et al.,2002). This data was collected with the aim of supporting rapid development of spoken dialogue sys-tems with advanced conversational capabilities.This valuable resource for research collected dialogues from8different systems:AT&T(Levin et al.,2000b),BBN(Stallard,2000),CMU (Rudnicky et al.,1999),COL(Pellom et al.,1999)(Pellom et al.,2001),IBM(Erdo˘g an, 2001),LUC(Potamianos et al.,2000),MIT(Seneff&Polifroni,2000)and SRI.Most of the systems adopt the frame-based architecture with some variations in the dialogue strategies, only the CMU Communicator adopted the agenda-based architecture.Figure2.1illustrates a dialogue from the MIT Communicator system.We can observe that manual design of this kind of dialogue is a labour-intensive task due to the many possible strategies that must be taken into account.In addition,manual design is prone to errors:for instance,in the second leg the system implicitly confirmed the user’s departure from“Zurich”(line31),but in line43the ASR recognizes a new departure city“Newark”.Here the dialogue manager assumed that the user changed his mind,which may be a reasonable action.In line 44the system explicitly confirms this new assumption,and the recognizer provides a wrong sequence of words due to a mixture of in-vocabulary words and out-of-vocabulary words(line 47).In line48,the system assumes a correct confirmation and the user is annoyed with this new departure city.At this point,a potential better strategy(considering dialogue history) may be to ignore this new assumption,possibly saving system turns,or simply minimizing user annoyance.Thus it is a fact that there will be some design issues that escape system developers,which makes automatic design of dialogue strategies an attractive option.Despite the previous research efforts in spoken dialogue management,more investigations are needed in order to build systems supporting successful,efficient and natural conversations. The following are some potential research gaps in this area:•To investigate methods for optimizing dialogue strategies exploiting system and users’knowledge•To balance the trade-off between userflexibility and system understanding•To separate the domain-independent and domain-dependent system features•To explore how dialogue managers can be abstracted and reused across domains2.1.Spoken Dialogue Management7LINE TURN UTTERANCE01. SYS: how may i help you?02. USR: thank you i want to fly from kansas city missouri on october third in the morning to denver colorado03. [thank you i want to fly from kansas city missouri on october third in the morning to denver colorado]04. SYS: okay, flights from kansas city to denver departing 5 a.m. and noon on Wednesday october third...05. USR: what else is available please06. [what are available please]07. SYS: i have a nonstop united flight leaving at 6:20 a. m. and arriving at 7:01 a. m would that work?08. USR: please09. [nowhere]10. SYS: i have a nonstop united flight leaving at 6:20 a. m. and arriving at 7:01 a. m would that work?11. USR: no i want to fly out later than six twenty12. [no i want to fly later in the morning]13. SYS: okay, flights from kansas city to denver departing after 6:20 a.m. on Wednesday october third...14. would that work? i also have a united flight leaving at 7:40 a. m.15. USR: i will take the united leaving at seven forty16. [i will take the united leaving at seven forty]17. SYS: united flight 1631 has been added to your itinerary. you depart... and arrive.... what date will you be returning on?18. USR: i want to leave denver on october fourth in the afternoon and i want to fly from there to zurich19. (switzer-) switzerland20. [i want to leave denver on october four in the afternoon and i want to fly american zurich switzerland switzerland]21. SYS: okay, flights from denver to kansas city departing between noon and 6:00 pm on thursday october22. fourth. one moment please. i have flights on...23. USR: no24. [no]25. SYS: i have a nonstop united flight leaving at 12:38 p. m. and arriving at 3:13 pm? would that work?26. USR: no i am flying from denver to zurich switzerland27. [no i am flying from denver to zurich switzerland]28. SYS: okay from denver to zurich. what date will you be travelling?29. USR: october four30. [october four]31. SYS: okay, from denver to zurich on thursday october fourth. can you provide the approximate departure32. time or airline?33. USR: i'd like to leave in mid afternoon34. [i would like to leave in the afternoon]35. SYS: okay, flights from denver to zurich departing between noon and 6:00 pm on thursday october fourth…36. USR: the american flight please37. [the american flight please]38. SYS: american flight 1078 connecting to american flight 170 in chicago o'hare ... what date will you be returning on?39. USR: i'm going to return to kansas on october fifteen40. [sunday returning in atlanta at four eighteen]41. SYS: okay, flights from zurich to kansas city departing at 4:18 pm on sunday october seventh… would that work?42. USR: (-o) i want to leave zurich in the late morning please43. [i want to leave newark in the late morning please]44. SYS: i heard you say you want to depart from newark. is that correct? if you need help, you can say,45. for example, what cities do you know in oregon.46. USR: i i don't want to go to oregon47. [i will what about oregon]48. SYS: okay, flights from newark to kansas city departing between 10:00 am and noon... would that work?49. USR: no i am not in newark i'm flying from zurich z. u. r. i c. h. switzerland remember you booked me to go to zurich50. [no i am not in newark on friday from new york and you are i see switzerland remember you booked51. me to voyager]52. SYS: sorry, i'm not sure what you said.53. USR: i want to leave zurich switzerland on on october fifteen in the late morning and fly back to kansas city54. [i want to leave zurich switzerland on october fifteen in the late morning and fly back to kansas city]55. SYS: okay, flights from zurich to kansas city departing between 10:00 am and noon on monday october fifteenth…56. USR: ...Figure2.1:Sample dialogue from the MIT Communicator system illustrating a three-legflightbooking:leg1from lines1to16,leg2from lines17to37,and leg3from lines38to56.Thetags SYS and USR correspond to system turns and user turns.The word sequences in squaredbrackets are the speech recognition results.8Chapter2.Previous Work•To compare the architectures in domains of different complexity•To extend the architectures for more complex tasks2.2Human-Computer Dialogue SimulationThe task of human-computer dialogue simulation consists of generating artificial conversations between a spoken dialogue system and a user.The aim is to automate the optimization and evaluation of spoken dialogue systems.Dialogue simulation may be performed at different levels of communication.The speech and word levels are useful for ASR systems in order to train acoustic and language models,and the intention level is useful for dialogue managers in order to train dialogue strategies.The following factors motivate the use of dialogue simulation for dialogue management:•Training optimal dialogue strategies requires many dialogues to derive an optimal policy and learning from real conversations may be impractical since it is expensive,labour-intensive,and time-consuming.An alternative is to use simulated dialogues.•Simulated dialogues can be used to evaluate spoken dialogue systems at early stages of development,and potentially to discover errors that may help to reduce expensive and lengthy trials with human subjects.•When a dialogue manager is updated,the previous optimization is no longer valid and another optimization is needed.At this point,simulated dialogues may help to speed up the development and deployment of optimized spoken dialogue systems.Several research efforts have been undertaken in this area and the following dimensions are used to summarize such investigations:1.Approach:Whilst some approaches are rule-based(Chung,2004)(Lin&Lee et al.,2001)(L´o pez-C´o zar et al.,2003),others are corpus-based(Eckert et al.,1997)(Scheffler &Young,2000)(Scheffler&Young,2001)(Georgila et al.,2005b).The advantage of the corpus-based methods is towards minimizing the portability problem(lack of expertise and high development costs)(Sutton et al.,1996).Rule-based methods tend to be ad-hoc for the task and domain.munication Level:Most of the investigations are intention based,and some usethe speech and word levels;depending on the purposes of the simulated dialogues.The investigations based on intentions have the purpose of optimizing dialogue strategies (Eckert et al.,1997)(Scheffler&Young,2000)(Levin et al.,2000a)(Scheffler&Young, 2001)(Pietquin&Renals,2002)(Georgila et al.,2005b).(L´o pez-C´o zar et al.,2003)use the speech and word levels in order to evaluate different speech recognition front-ends and dialogue strategies.(Chung,2004)uses the speech and word levels in order to train speech recognition and understanding components.3.Evaluation:A few investigations attempt to evaluate the simulated dialogues.(Eckert etal.,1997)(Scheffler&Young,2000)(Scheffler&Young,2001)use the average number of turns.(Schatzmann et al.,2005a)use three dimensions:high level features(dialogue and turn lengths),dialogue style(speech-acts frequency;proportion of goal-directed ac-tions,grounding,formalities,and unrecognized actions;proportion of information pro-vided,reprovided,requested and rerequested),and dialogue efficiency(goal completion rates and times).Finally,(Georgila et al.,2005b)use perplexity and a performance function based onfilled slots,confirmed slots,and number of actions performed.。