Usability

D9d

Proposal for exemplars and work towards them:

Usability

Workpackage 9 Deliverable

Date: 30th November 2005

IST project contract no. 507422

Project title HUMAINE

Human-Machine Interaction Network on Emotions Contractual date of delivery November 30, 2005

Actual date of delivery November 30, 2005

Deliverable number D9d

Deliverable title Proposal for exemplars and work towards them:

Usability

Type Report

Number of pages 30

WP contributing to the

WP 9

deliverable

Task leader KTH

Author(s) See page 5

EC Project Officer Philippe Gelin

Address of lead author:

Kristina H??k

Department of Computer and Systems Sciences

Stockholm University/Royal Institute of Technology (KTH)

Forum 100

164 40 Kista

Sweden

Table of Contents

1THE PLACE OF THIS REPORT WITHIN HUMAINE (5)

2BRIEF OVERVIEW OF WORKPACKAGE 9 AND THE EXEMPLAR PROPOSAL (6)

2.1The field covered by Workpackage 9 (6)

2.2The research objectives (7)

2.2.1 Main elements of the exemplar (8)

2.2.2 How the elements link to each other (8)

2.2.3 How the elements link to other aspects of HUMAINE (9)

3THE PLANNED PROGRAM OF RESEARCH (10)

3.1Element 1: Criteria for usable affective interaction systems (10)

3.1.1 Main participants (10)

3.1.2 Main steps planned towards producing element 1 (10)

3.2Element 2: Evaluation metrics for criteria (11)

3.2.1 Main participants (11)

3.2.2 Main steps planned towards producing element 2 (11)

3.3Element 3: Existing user - centred methods for design and evaluation (12)

3.3.1 Main participants (12)

3.3.2 Main steps planned towards producing element 3 (12)

3.4Element 4: New methods for design and evaluation (14)

3.4.1 Main participants (14)

3.4.2 Main steps planned towards producing element 4 (14)

4RESEARCH PROGRESS TO DATE (17)

4.1Element 1: Criteria for usable affective interaction systems (17)

4.2Element 2: Evaluation metrics (17)

4.3Element 3: Existing user - centred methods for design and evaluation (19)

4.4Element 4: New methods for design and evaluation (22)

5CONCLUSION (27)

5.1Relation to the state of the art (27)

5.2Evidence of esteem (27)

6REFERENCES (29)

1The place of this report within HUMAINE

The HUMAINE Technical Annex identifies a common pattern that is followed by most of the project’s workpackages

The measure of success will be the ability to generate a piece of work in each of the areas which exemplifies how a key problem in the area can be solved in a principled way; and which also demonstrates how work focused on that area can integrate with work focused on the other areas. We call these pieces of work exemplars. The exact form of an exemplar is not prespecified: it may be a working system, but it might also be a well-developed design, or a representational system, or a method for user-centred design. (p 4)

To that end, each thematic group will work out a proposal for common action, embodied in one or more exemplars to be built during the second half of the funding period (p.16)

The process will begin with production by each thematic group of a review of key concepts achievements and problems in its thematic area; and drawn from the review, an assessment of the key development goals in the area. This review and assessment will be circulated to the whole network for discussion and comment, aimed both at building understanding of basic issues across areas, and at identifying the choices of goal that would be most likely let the different groups achieve complementary developments. That consultation phase will provide the basis for deliverables in month 11, which describe in some detail a few alternatives that might realistically be chosen as exemplars in each area, and their linkages to issues in other thematic areas. A decision and planning period will follow, involving consultation within and between thematic areas, leading to presentations at the second plenary conference, which will describe a single exemplar that has been chosen for development in each area, and the way work on the exemplar will be divided across institutions. The remainder of the project will be absorbed in developing the chosen exemplar. (p. 21)

The consultation phase has now ended. Near-final plans were presented to the whole network at the Plenary in May 2005, and adjustments have been made accordingly.

This deliverable reports the plans that have now been set out for the remaining 25 months of the project. They are necessarily provisional, because they will be subject to two reviews (in 2006 and 2007) before they are completed.

Work has begun on several aspects of the planned programme. It is also reported.

Ethical issues affect the whole of HUMAINE, but rather than repeating essentially similar points in multiple deliverables, they will be handled coherently in a single document, D0o (Science and Society).

The following persons have contributed to the work reported in the deliverable:

Kristina H??k, Jarmo Laaksolahti, Fiorella de Rosis, Brigitte Krenn, Katherine Isbister, Benoit Morel, Asimina Vasalou, Catherine Pelachaud, Markus Ballegooy, Stephen Westerman, Tanja B?nziger and WP9 members

The institutions that have contributed are:

KTH, OFAI, Imperial College of Science, Technology and Medicine, University of Paris 8, Università Degli Studi di Bari, T-Systems Nova GmbH, La Cantoche Production, LEEDS, GERG

2Brief overview of Workpackage 9 and the exemplar

proposal

While the rest of HUMAINE will produce knowledge and theories of many different aspects of emotional systems (ECA behaviours, emotion recognition, interaction principles, exemplar databases, and emotion theory) the overall goal of WP9 is to use that knowledge to produce design and evaluation methods able to deliver functioning affective end-user applications. Such applications allow users to become affectively involved in the course of interaction with the system.

Our strategy for working within an end-user application framework to evolve the usability of affective systems can be divided into two focus areas:

1.Ends: Determining qualities and criteria that demark emotional systems as usable and

evoking the desired experiences for users.

2.Means: Forging process—design methods, project goals and evaluation strategies —

that will steer a project towards producing a ‘successful’ affective application.

The WP9 exemplar is designed to develop, explore and determine key criteria and methods for designing and evaluating affective systems

2.1 The field covered by Workpackage 9

The area covered by this workpackage is described in the Technical Annex, particularly in Section 6.2, and in more depth in the review and assessment document for the workpackage. We summarise the area here so that the deliverable can be read as a stand-alone document. Regarding Ends, we do not believe there are simple, replicable correlations between system properties and all end users’ experiences. The experience of using an emotional system is not a property of the system itself, but rather is something that arises in the interaction between user and system. To quote Sengers and colleagues (2004):

“Rather than experience as something to be poured into passive users, we argue that users actively and individually construct meaningful human experiences around technology. They do so through a complex process of interpretation, in which users make sense of the system in the full context of their everyday experience.”

Along the same lines, Suchman has criticized the cognitivistic reductionist basis for the field of affective computing (Suchman, 2002), reducing human emotional responses to discrete, universal component parts, arising from some underlying state, and offers examples of alternative designs that put emotional interaction at the core but where: “we might differently conceptualise affective encounters at the interface: as irreducibly contingent meetings of particularly situated persons with equally particular, dynamic, and culturally inflected things.” As we discuss below, usability of emotional systems and the process of arriving at functioning emotional systems need to be sensitive to how end-users are part in co-constructing the experience and be part of this dynamic and cultural process.

Usability traditionally focuses on goals such as effectiveness, efficiency, safety, utility, learnability, and memorability. These objective usability goals contrast with user experience goals, which cover subjective qualities such as being fun, rewarding, motivating, satisfying, enjoyable, and helpful. Usability goals and user experience goals often stand in complex relationships, involving tradeoffs such as safety vs. fun or efficiency vs. enjoyability (Preece et al., 2002). Introducing emotion thus raises many new dimensions for research on usability to address. It becomes, for instance, a serious issue whether users feel a system is ‘sympathetic’ or morally acceptable, whether it engages them emotionally. Furthermore, emotions address directly inherently adaptive faculties of humans, posing challenges for methods of user studies and artefact design.

Work Package 9 will take a primarily qualitative, situated/contextual approach to measuring Ends. A sub-group of Work Package 9 may also focus on evaluating and developing measures aimed at isolating meaningful evaluative variables and testing various components of systems against these variables.

Regarding Means, we believe that evaluation of affective systems is vital not just at the end of the design process, but as an integral part of the design process from the beginning. Having the ability to bounce early intuitions and design sketches off of real users can make key contributions to the evolution of a truly engaging end application, and may even inform the affective theory that led to the application itself.

Work in the field of HCI has made great strides toward merging design and evaluation of productivity-oriented systems (see the goals section of this document for some relevant citations); we seek to extend and adapt participatory design methods and approaches to the needs and desired outcomes of affective system projects. For example, encouraging nonverbal participation and evaluation, and using physical prototypes and fake systems operated by humans. We believe this holistic and integrated approach will not only lead to better system designs, but may also provide important ‘in situ’ insights back to affective theory makers. Thus, it is not the primary intent of WP9 to test computational emotion theories or evaluate how particular parameters for e.g. raising an eyebrow in an ECA should work. Instead, the task of WP9 will be to produce design process and evaluation theory, methods, and measures needed to take all that knowledge into the design process and through it, to produce emotional systems that will engage with end-users to create compelling experiences.

2.2 The research objectives

The exemplar proposed for WP9 is a Framework for Design and Evaluation of Usable Affective Interaction Applications. Similar to how HCI has benefited from user-centred design processes, our aim is to test and further develop such methods for the area of affective interaction. This in turn will hopefully push the research frontier forwards and serve as an important exemplar to others in the field. The framework contains an assortment of techniques and methods for various aspects of the design process including generating initial ideas for affective applications, refining ideas into systems/products or validating the affective interaction loop in finished systems/products. As a whole the framework is intended to be a broad resource for designing and evaluating affective systems – representing first and foremost a user-centred perspective on affective interaction applications. The tools and methods of the framework will be applied and evaluated in a variety of situations/applications/domains. Records from the sessions will provide valuable guidance for future users of the tools and methods regarding their proper usage and expected results.

2.2.1Main elements of the exemplar

The exemplar consists of four main elements.

Criteria for usable affective interaction systems

Develop an understanding for what makes affective interaction systems successful and formulate some criteria. These criteria will not be objective, independently measurable entities, but will make sense relative to the specific application domain, aim to capture subjective experiences of the application, and foremost, be related to the designer’s intention for the application.

Evaluation metrics for criteria

Translate criteria for affective interaction systems into usable metrics and methods for eliciting them.

Existing user-centred methods for design and evaluation

Condense experiences from applying existing user-centred design methods to the design and evaluation of affective interaction systems – which methods work and which do not? Examples of such methods include body-storming (Oulasvirta et al, 2003), interaction re-labelling (Djajadiningrat, et al., 2000) and personas (Cooper, 1999).

New methods for design and evaluation

Develop new methods for capturing unique aspects of affective interaction that can be used during an iterative design-evaluate-redesign process. Methods that will be investigated include: a sensual method for non-verbal mediation of affective state, a Wizard-of-Oz environment for multimodal emotional interaction, and an extended think-aloud protocol designed to capture emotional interactions.

2.2.2How the elements link to each other

Design and evaluation is a broad area, participants have differing interests and backgrounds, different use-cases require different methods, resources, and study angles. We are interested in the whole design-evaluation process (see Figure 1) and the subtasks of WP9 address different aspects of the process. The exemplar can still be viewed as a single piece though, since it explores one and the same idea: that a user-centred perspective will help focus the design process and create final application systems that involve users affectively. This exemplar will be a touchstone for integrating this perspective throughout the whole life-cycle of a system development process. While we strongly believe that this is the best kind of exemplar for WP9, there are still many white spots on the map. We do not know how to capture subjective experiences and affective involvement in such a way that it can provide feedback into the design process. A user-centred design perspective and development philosophy is most probably a good path to explore, but little is, as of yet, known in what to design for (flow, pleasure, fun, experiences, excitement, fear), and how to best make users involved.

Figure 1User-centred design framework

Positioning the subtasks within the user-centred design framework above ensures that subtasks are working towards a common goal. The nature of the framework with its different methods and tools all sharing the same conceptual framework also helps in this respect. The four subtasks follow on one-another. Once criteria for the success of a specific application project are established, there has to be corresponding measurements to know whether the design goals are met according to the same criteria. The process to ensure such a design and evaluation cycle in turn needs methods. All these tasks together will help shape the framework produced by this WP.

2.2.3How the elements link to other aspects of HUMAINE

WP9 cannot be separated from the rest of HUMAINE because evaluation depends on systems (or concepts) to evaluate. It can be seen in section 4 that there has been sustained interaction with teams in most areas of the network. The WP9 workshop, which will take place in January 2006, is designed to strengthen these collaborations. The main kinds of interaction can be summarised briefly as follows.

WP3: Methods need to be grounded in theory. WP9 absorbs and builds upon current affective theory to create appropriate evaluation instruments.

WP5: The annotated databases are expected to provide input to design processes in WP9, and will also serve as material in evaluation processes.

WP6: By working with Work Package 6 in the design of an Affective Interactive Embodied Conversational Agent system, we hope to test and refine our evaluation techniques and instruments, which will serve as our final exemplars. Moreover we plan to apply the representation language defined in WP6 for the WoZ-environment and also adopt it to the “sensual” method.

WP8: We may start a similar collaboration as with WP6 on specific application examples where we can apply our framework for design and evaluation.

3The planned program of research

3.1 Element 1: Criteria for usable affective interaction systems

3.1.1Main participants

Institution Person(s)

OFAI Brigitte Krenn

Università Degli Studi di Bari Fiorella de Rosis

KTH Kristina H??k, Jarmo Laaksolahti

3.1.2Main steps planned towards producing element 1

Subtask Carried out by Start / end

dates

OFAI, BARI M0/M48 Compilation of a commented literature list on ECA

evaluation studies and of initial definition criteria for

empathic ECAs.

KTH M0/M48 Explore success criteria for affective interaction wrt the

specific design aims.

The first task aims at a commented compilation of literature on ECA evaluation. As a starting point, the issue is addressed from three different perspectives: 1) Dimensions of evaluation, i.e. what aspects or conceptual dimensions of ECAs are evaluated, for instance: ECAs as social actors, ECAs as emotion expressing entities, ECA applications and their emotional effect on the user, etc. 2) Evaluation paradigms addressing the type of study, for instance: 3rd party observation, i.e. the user/subject is only observer of pre-fabricated scenes played by one or more ECAs; Wizard-of-Oz, i.e. the system is merely mock-up and the system behaviour is controlled by a human (the wizard); full user involvement with a fully automatic system, i.e. the user interacts with a fully computerized system. 3) Evaluation methods: While 1) and 2) address macro-levels of evaluation, 3) is concerned with micro-level aspects of evaluation, i.e. the individual strategies and methods employed in different evaluation experiments, such as quantitative and qualitative methods, questionnaires, semantic differentials, the specific set up of study plans and the such.

This contribution initially has been conceived as a literature survey terminating at month 19. In the course of the work in HUMAINE it has turned out that the task is better conceived as work in progress which is open during the runtime of the network and ideally stays active beyond its runtime, reflecting, complementing and continuing the work in HUMAINE.

In the second task we aim to explore success criteria for affective interaction application wrt the specific design aims of an application and what the designer intended to achieve. Only if the designer’s intentions are experienced by the end-users as intended is it possible to talk about a successful system. In addition more general concepts like flow, ambiguity, pleasure and expressivity will be explored as potential criteria for success. This will mainly by accomplished through a series of workshops, brainstorming sessions and practical application to use cases (se below). Our aim is that this will lead to an iterative refinement of useful criteria for developing affective interaction systems. A first workshop was held at the CHI conference in April 2005 and will be followed by a brainstorming session and the WP9 workshop in January 2006.

3.2 Element 2: Evaluation metrics for criteria

3.2.1Main participants

Institution Person(s)

KTH Kristina H??k, Jarmo Laaksolahti

OFAI Brigitte Krenn

Università Degli Studi di Bari Fiorella de Rosis

3.2.2Main steps planned towards producing element 2

Subtask Carried out by Start / end

dates

OFAI/BARI M13/M48 Construction of a collection of design, test and evaluation

strategies and methods for empathic ECAs.

Assessment of subjective evaluation metrics KTH M0/M30 The first task aims to give a review of the state-of-the art in evaluation studies on ECAs. In particular, the review will start from a reflection on the concept of ‘empathy’, to critically reflect on whether and how ECAs have proven, so far, to be able to establish this kind of relationship with their human users. The study will consider the evaluation criteria and methods applied, and will critically present in a flexible and dynamic structure the results produced. Results from this task feeds into “Criteria for affective interaction systems” and “Existing user-centred methods for design and evaluation”. The studied literature will also appear in the commented literature list described in task 1 under 3.1.2.

The second task deals with subjective evaluation metrics. A subjective evaluation metric can be quite open-ended – any sources of information may be collected and interpreted vis-à-vis the goals of the system. Anecdotal evidence, informal chats between users and system-

builders, tiny study sizes, forms structured to influence user interpretation, no discussion or analysis of results: this may sound like a to-do list for bad evaluation. But these choices are deliberately and thoughtfully made, highlighting underlying conceptual problems in using standard so-called ‘scientific’ evaluation techniques to evaluate emotionally involving systems. The subjective approach means that we might not be building systems for ‘normal’ or ‘average’ users. Instead we are interested in the richness and complexity of unique, individual users, cultural contexts, and resulting variety of interpretations and experiences of our systems. It is not appropriate to summarize the results of a study into a few statements that are said to hold for everyone. Also, the statistical averaging and laboratory simplifications necessary for reliable scientific statements may wash out all the details that interest us in a user-centred design process where we want input for the next cycle of design. We may prefer a rich, narrative, and singular understanding to a simpler but rigorous and generalisable understanding in the early stages of design. The eMoto study and the Sensual Evaluation Instrument (see below) are examples of an evaluation study and - method respectively where subjective metrics are essential.

Subjective measurements will be complemented by more generalisable and objective measures where appropriate and needed in order to further our knowledge of what constitutes a successful design. This will mainly by accomplished through a series of workshops, brainstorming sessions and practical application to use cases (se below). Our aim is that this will lead to an iterative refinement of useful metrics for evaluating affective interaction systems. A first workshop was held at the CHI conference in April 2005 and will be followed by a brainstorming session and the WP9 workshop in January 2006.

3.3 Element 3: Existing user - centred methods for design and

evaluation

3.3.1Main participants

Institution Person(s)

La Cantoche Production Benoit Morel, Justine Plessier

T-Systems Nova GmbH Markus Ballegooy

University of Paris 8 Catherine Pelachaud

KTH Kristina H??k, Jarmo Laaksolahti

LEEDS Stephen Westerman

3.3.2Main steps planned towards producing element 3

Subtask Carried out by Start / end

dates

WP9 Brainstorming session on generating application ideas using existing design and evaluation methods KTH, Paris 8, Cantoche, OFAI

M25/M25

Application of existing user-centred methods to use-cases KTH, Paris 8,

Cantoche, T-

Systems

M0/M48

Develop a taxonomy of affective systems usability testing LEEDS M24/M33 In conjunction to the WP9 workshop to be held in Stockholm, a brainstorming session will be organised. The session will be devoted to exploring how to use existing methods for design and evaluation to generate usable affective application ideas. Participants will include industrial as well as academic partners from HUMAINE.

Throughout the project methods for user-centred design and evaluation will be applied to the use-cases proposed in D9c (e.g. the eMoto application described below), as well as other applications, and experiences will be gained regarding their suitability for affective interaction design. For instance, until now, for every application to be developed, Cantoche meets their clients in order to understand why an ECA should be used and what is the most appropriate one. Together, they define:

- The mission for the ECA ( e.g. Expert, tutor, assistant).

- The target: what kind of users will interact with the ECA (culture, gender, age)

- The scene: what is the support where the ECA will act (mail, web site, presentation) All along the development of the application, the client tests the system and provides feedback that helps to refine it before the application is launched to real users. So, our first users are the clients. Within WP9, we aim to bring na?ve users in the development process and evaluate what users centred methods can bring us.

Within WP6, Paris 8 works on capabilities required to build an affective interactive ECA. Some cover low level aspects of ECA specification (behaviour expressivity factor, multimodalities coordination) other deal with the definition of means to show level of engagement through displaying backchannels, or to endow agent with visual attention and perception. While in WP6 we are concentrating in defining the concepts and providing some forms of implementation of these capabilities, no work is undertaken to understand if and when these capabilities are necessary and important for a given application. For example, we would like to know to which degree of subtlety ECA behaviours should be specified: e.g., to which degree it is important for the application to have an ECA able to display facial expressions of blend of emotions. We aim to use the outcome of user centred studies to understanding these issues and be able to refine which capabilities are required to build an affective interactive ECA in the context of an application.

LEEDS will develop a simple taxonomy of affective systems that can be used as a basis for the examination of usability issues. This taxonomy classifies on the basis of direction of emotional communication and requirements for elicitation and assessment of user affect. Existing systems and empirical evidence will be reviewed in this context, with specific reference to usability-related outcomes, and implications for ‘traditional’ usability testing methods.

3.4 Element 4: New methods for design and evaluation

3.4.1Main participants

Institution Person(s)

KTH Kristina H??k, Jarmo Laaksolahti, Katherine

Isbister

OFAI Brigitte Krenn

Università Degli Studi di Bari Fiorella de Rosis

Imperial College Asimina Vasalou

LEEDS Stephen Westerman

3.4.2Main steps planned towards producing element 4

Subtask Carried out by Start / end

dates Sensual evaluation instrument KTH M0/M48 - Design of the sensual evaluation instrument KTH M0/M36 - Application/evaluation of the instrument KTH M18/M48 The first task will investigate the possibilities for subjective self-report of affective state in nonverbal ways. As described above this work has already started and resulted in one possible toolset. This toolset will iteratively be refined and expanded during the project.

The second task deals with applying the toolset in “real-world” settings to verify its functionality and to gain a better understanding of possibilities and limitations of the proposed form of evaluation studies. Currently a version of the toolset has been constructed. In the next phase of development, we plan to use the Sensual Evaluation Instrument with designers who are seeking affective feedback from users about their work, with preference given to HUMAINE members' projects. We will include the SEI in the WP9 Workshop in January in Stockholm as a tool for consideration and use. We hope that testing the tool as a design aid with evolving systems will validate the SEI's helpfulness, as well as lead to evolution of the tool itself.

Subtask Carried out by Start / end

dates

WoZ test bed OFAI/BARI M0/M48

- Implementation of the WoZ tool BARI M0/M19

- Realizations of the design-evaluation loop OFAI/BARI M13/M48

The first task aims to produce a tool which enables performing evaluation studies on ECAs in different conditions: different ECAs, different domains, and different evaluation criteria. This tool will enable performing a comparison of results that may be obtained by means of ‘subjective’ (or cognitively-based) methods and by means of ‘behavior observation’ methods: therefore, a reflection on ‘methods’ and evaluation criteria’.

The second task concerns design and conduction of WoZ experiments, in order to hypothesize and test aspects of empathic interactions of users with ECAs. These experiments will consist in sets of small-scale iterative studies, which will be designed so as to provide to users a framework which will enable them to express their requirements with a relatively high degree of freedom. Results expected from these experiments will not only consist in statistics about subjective and objective evaluation parameters, but will provide, as well, a corpus of data upon which to ground the design of artificial dialogs. The insights gained from every WoZ experiment will therefore feed into the redesign of the WoZ experiments, as well as into modelling of fully computer generated empathic ECAs, the effect of which will then be tested in user experiments. Results from this task feeds into “Evaluation metrics for criteria” and “Exploring new methods for designs and evaluation”. OFAI, in particular, will make use of the WOZ setting in order to explore and develop methods and strategies suitable for rapid prototyping of an application where a human user and an ECA collaborate in a quiz game. This work will be done in the context of an FP6 Cognitive Systems project that is about to start beginning of 2006.

Subtask Carried out by Start / end

dates

Extended think aloud protocol Imperial, GERG M27/M31

- Development of affective applications Imperial, GERG M27/M28

Imperial, GERG M28/M29 - Development of the extended emotional-think aloud

protocol and pilot study

- Cross Cultural think aloud study Imperial, GERG M29/M31

We are currently developing an affective application that will serve as a platform for our experiment. Our objective is to determine whether the think aloud protocol can reveal emotional data. For the purposes of our study, we are developing an application that will support a short set of tasks that include three emotion inducing events, one for joy, one for anger and one for embarrassment.

We will pilot a few think aloud prompts. In one concept we will prompt users with a flickering light which will light up their screen when they are silent for too long. In another concept we will inform users that the facilitator is prompted by the system on silent moments and speaks out during those moments. What we hope to gain from this pilot is an

understanding on which concept is more personable and less alienating to the user. We will also use questionnaires to access to which extent the 2 criteria of interference and suppression are being affected.

Once we choose a prompting system we will conduct the experiment. In this experiment, each user will be trained on the think aloud protocol. Following training, he/she will interact with each of the three emotion-inducing applications. The verbalisations will be recorded and then analysed and rated by na?ve, independent experts. The study will be conducted in the UK and in Switzerland in anticipation and detection of cultural differences in vocal expression. We do not expect our sample to exceed 30 users (15 for each country).

Subtask Carried out by Start / end

dates Psychometric assessment model/approach LEEDS M24/M30 - Further testing of the model LEEDS M24/M28 - Revision of the model LEEDS M28/M30 LEEDS will be beginning further tests of their model/assessment approach in month 24. Results relevant to HUMAINE are expected to be available in month 28. Further revisions to the model/assessment method are expected to be available in month 30.

4Research progress to date

4.1 Element 1: Criteria for usable affective interaction systems Progress on the “webpage consisting of a commented literature list on ECA evaluation studies”: The literature that has been collected so far is considered as a seed for a “literature collection in progress”. A first version of the commented and structured list will be made available on the HUMAINE portal by the end of 200

5. For a brief description of the initial structure of the collection see section 3.1.2. This list will be open for additions of literature as well as supplementary comments on existing entries and alternative structuring of the entries constituting the seed. Ideally a single literature entry will be embedded in a web of links and be annotated with a number of comments referring to different aspects of ECA evaluation. Exploring criteria for affective interaction systems is mainly accomplished through a series of workshops, brainstorming sessions and practical application to use cases. Our aim is that this will lead to an iterative refinement of useful criteria for developing affective interaction systems. A first step was the workshop on evaluation of affective interfaces was organised at the CHI (Computer-Human Interaction) conference in Portland, Oregon. The workshop was named “Evaluating Affective Interfaces: Innovative Approaches” and 12 papers were selected for presentation at the workshop. The researchers at the workshop presented work that spanned a large range of theoretical perspectives and thereby quite a large span of possible methods for evaluation. In our perspective, this shows the diversity of what is meant by affective interactive systems and also how different researchers view what can be seen as successful design/criteria. The CHI workshop will be followed up by the WP9 workshop in January 200

6. Prior to the workshop a brainstorming session will be held that explores how to generate application ideas using existing design and evaluation methods. The session and the following workshop will include researchers, designers and practitioners. These groups all have their own views on what is important in a design process. For researchers the most important thing is to prove the success of their systems using some suitable (usability)criteria. Designers on the other hand search for methods that support and provide feedback to the design process. Practitioners search for good examples that illustrate ideas in action and that can provide practical advice on how to build applications. The list of problems collected for deliverable D9a also shows the diversity of possible criteria. In order to gain a complete picture of affective interaction design and evaluation, all of the above viewpoints need to be taken into account and their relationships explored.

4.2 Element 2: Evaluation metrics

One of the main goals of HUMAINE is to make different theoretical strands within the large field of affective computing to get together and debate their different viewpoints. The aim is to enable fruitful new collaborations between fields that may previously have been more or less unaware of each other. In WP9 one such theoretical stance debate has been brought up several times. In short summary it is a meeting between Human-Computer Interaction goals and methods and more psychology-inspired goals and methods. For example, we debate the purpose behind why we do user studies, how to do them, and which methods are "scientific" and which are not.

In HCI, researchers are turning more and more towards what we could name "ecological validity". That is, systems needs to be brainstormed, developed and tested in settings that are

very close to the "real" settings they will be used in. We talk about studies "in the wild" and issues like "staged lived experiences", that is, how we can set up situations with half-working systems that resemble the real situations in which they will be used.

Theoretical foundations from this movement is picked from phenomenology (see e g Paul Dourish' book "Where the action is"), from ethnographic work, from grounded theory, from situated cognition (see Suchman "Plans and situated action"), from activity theory (see Nardi "Context and Consciousness: Activity Theory and Human-Computer Interaction”), participatory design (or the so-called Scandinavian school of design), and so on. In common for all these is an emphasis on moving beyond cognitivism and instead try to see the whole context in which a system is used, the other tools people make use of, their daily practices, their culture, etc. The belief is by understanding more of the whole complexity and being more “emphatic” with the end users, actually placing ourselves as designers in their shoes so to speak, we will be able to produce better systems.

In our field, affective interactive systems, we have additional problems compared to more traditional HCI-problems with the development with tools. We are trying to design for some kind of user experience. We cannot guarantee that the users will experience something because it is obviously constructed by themselves, in their minds, but we can design more or less good interaction that at least form the basis for certain kinds of experiences. All we can do is a more or less good job in the design process. The problem lies in how we evaluate our systems so that we can provide the designer of the system with the right level of feedback? What will help the designer to figure out where the weaknesses in the system are and what the best way of altering it is to actually reach the design goals? As the experience has elements of unconscious processes (both cognitive and bodily) we need to find some way of tapping into those. In addition, we also need to compare those unconscious emotional reactions to the more conscious reflections that users make. Both are relevant in the design process. I will not buy a system that does not fit with my attitudes and beliefs about the system.

We therefore need two kinds of criteria: what are the reasons that customers will pick an application off the shelf in the store or on the web and decide to use it? And second, once they do indeed use it, what are the criteria for measuring that their reactions to it are indeed the ones that the designer aimed for?

In the more psychology-oriented methods, it becomes vital to do more controlled lab studies in order to single out which variables do indeed contribute to the success of an interaction with an affective interactive system. They typically require statistically significant results and ways of isolating what gives rise to what.

Most cognitivists, psychologists, lab-researchers are well aware of validity issues in general, including ecological validity. But they are facing a trade-off between preserving the complexity of "ecological" situations, on one hand, and experimental control, on the other hand. It is not infrequent that "ecological validity" (or rather complexity) is sacrificed for the benefit of studying specific behaviours in isolation. The inconveniences of isolating any psychological process from its context are well known, nevertheless it is often impossible to test hypotheses or make systematic observations in fluctuating contexts that might differentially affect the processes under investigation. Aspects of the context that are likely to influence the investigated processes must be controlled and/or systematically manipulated to allow researchers to understand their effects.

The problem for the emotion researcher lies in the following: Emotions are reactions that include several components (facial/vocal/gestural/postural expressions, physiological

reactions, feelings, they are tightly linked to attentional and motivational processes, memory, decision making, etc…). One important component is the subjective feeling associated to the emotional reaction. It belongs to the domain of the subjective experience of the person and can only be accessed through subjective reports. We have ways to measure the other components of the emotional reactions without asking the person to give a report, but if we want to know what they feel we can relay only on subjective reports (there is no alternative method).

Both these viewpoints and methodological perspectives need to co-exist side by side in the work in WP9, and they can indeed inform each other, but we also need to recognise that they come from quite different theoretical foundations and will therefore contribute to our knowledge of what designs work in quite different ways.

A major problem in HUMAINE is that the different WPs have quite different goals. The goal of, for example, WP3 is to generate emotion theory in general. The goal of WP9 is to find the methods that will generate good applications. These are quite different goals. And they require different methods. An evaluation method in WP9 by necessity has a range of extra demands that it needs to fulfil, such as being cost-efficient, not avoiding the complexity of the future use situation, etc. Otherwise it will never be used by the people who are supposed to develop affective interactive applications in the future.

Thus, the discussion on evaluation metrics is in a very lively state in WP9 where these two theoretical strands meet. One strand argues that isolating variables in the lab is crucial in order to know what we are measuring and whether indeed our affective interactive systems form the basis of the kinds of experience we want users to experience. Another strand is more interested in a phenomenological strand where evaluation is seen as an on-going process between users and designers where the whole story needs to be taken into account. We are certain that there will be a lively debate at the upcoming WP9 workshop on these issues.

4.3 Element 3: Existing user - centred methods for design and

evaluation

Several evaluation studies have been performed, or are planned, that explore the use of existing methods for user-centred design albeit sometimes in novel ways. Work has also been started by LEEDS on a taxonomy of affective systems that can be used as a basis for the examination of usability issues.

A wild evaluation of users’ emotional engagement

eMoto is a mobile service for sending and receiving affective messages, with the explicit aim of addressing the inner experience of emotions. In eMoto a user first writes her text message and then finds a suitable affective graphical expression to add to the background of her text. To find these expressions, she navigates in a circular background of colors, shapes and animations through using a set of affective gestures. Following the two-tiered evaluation model (H??k, 2004) we first verified that users understood and could interpret each of these parts of eMoto before we combined them and set up a user study where we looked for results on the overall goal, which was having users emotionally engaged in an affective loop.

Figure 2 Users' probes

Figure 3 Friends' probes To determine whether someone is emotionally engaged and moreover emotionally involved in an affective loop, is however a very difficult task. In our final evaluation of eMoto we went for a combination of Isomurso and colleagues’ following a friend method (Isomurso et al. 2004) and the cultural probe method first described by Gaver and colleagues (Gaver et al. 1999). Both these methods contain aspects of letting users themselves become the researchers. In this final evaluation we wanted users to experience eMoto in the wild and not in the more staged lab environment where we believe it is hard for us to create a setting where users can become truly emotionally engaged, but also the wild, would not be that much wild, if we researchers came along. Instead we gave users material for them themselves to document their experiences (see Figure 2), but also we prepared a bag of material for them to give to a close friend of theirs (see Figure 3). eMoto is very much inspired by emotional body language, a language in which friends understand each other far better than we as outsiders understand them. Therefore we believe friends can be better, than at least computer scientists, performing the task of analyzing users’ emotional engagement.

After having received all the material from the users we quickly looked through all the films, diaries and postcards from both users and their friends and together with our log data formed a quick and dirty analysis of each user’s experiences. We then discussed this analysis with our users during individual interviews. The intention was to get a starting point for the discussion but still stay to the notion of letting the users become the researchers.

We have just started to go through these interviews but preliminary results indicate that we in our design of eMoto have succeeded in presenting users with a foundation of expressiveness that carries shared meaning but opens up for individual expressiveness and emotional engagement.

Evaluation of an emotion aware voice portal

One of the major problems in evaluating affective systems is to produce or find usage situations where real user emotions emerge spontaneously without an artificial add on (artificial laboratory situations). In T-System’s evaluation study we try to obtain “quasi – natural” usage scenarios by deceiving the subjects about the tests purpose.

SUSY is an emotion aware voice portal where customers can extend their contract with the operator and select a new mobile phone from a list. The usability of the complex selection process is the main subject of the study, also the general acceptability of “speech directed upgrading” is a question we want to clarify. Since there are many ways to get into problems during the selection process, we added an emotional component to the system. Within dialog situations where the user gets angry and anger is detected by the acoustic classifier the system