Introducing evaluation

Introducing evaluation

1. What, why, and when to evaluate

   • Users want systems that are easy to learn and to use as well as effective, efficient, safe, and satisfying.
   • Being entertaining, attractive, and challenging, etc. is also essential for some products.
   • what
     • focus on users and their tasks
     • observe, measure, and analyze their performance with the system
     • design iteratively
   • why
     • Problems are fixed before the product is shipped, not after.
     • The team can concentrate on real problems, not imaginary ones.
     • Engineers code instead of debating.
     • Time to market is sharply reduced.
     • Finally, upon first release, your sales department has a rock-solid design it can sell without having to pepper their pitches with how it will all actually work in release 1.1 or 2.0.
   • when
     • Evaluations done during design to check that the product continues to meet users' needs are know as formative evaluations.
     • Evaluations that are done to assess the success of a finished product, such as those to satisfy a sponsoring agency or to check that a standard is being upheld, are know as summative evaluation.

2. HutchWorld case study

   • is a distributed virtual community developed through collaboration between Microsoft's Virtual Worlds Research Group and librarians and clinicians at the Fred Hutchinson Cancer Research Center in Seattle, Washington.
   • The system enables cancer patients, their caregivers, family, and friends to chat with one another, tell their stories, discuss their experiences and coping strategies, and gain emotional and practical support from one another
   • A common approach to evaluation in a large project like Hutchworld is to begin by carrying out a number of informal studies.
     • Typically, this involves asking a small number of users to comment on early prototypes.
   • early design ideas
     • what is the typical treatment process, what resources are available to the patient community, and what are the needs of the different user groups within this community?
     • They had to be particularly careful about doing this because many patients were very sick.
     • The development team decided that HutchWorld should be available for patients any time of day or night, regardless of their geographical location
     • Before testing could begin, the team had to solve some logistical issues. There were two key questions. Who would provide training for the testers and help for the patients? And how many systems were needed for testing and where should they be placed?
   • test
     • Test 1
       • What was learned?
         • This V-Chat test brought up major usability issues. First, the user community was relatively small, and there were never enough participants in the chat room for successful communication-a concept known as critical mass.
         • The team also observed that the users used the computers to play games and to search the web for cancer sites approved by Hutch clinicians.
       • Early observations onsite
         • In the informal test at the hospital, six computers were set up and maintained by Hutch staff members.
         • The team observed the usage of the space during unscheduled times, and they also observed the general usage of the prototype.
     • The redesign
       • Based on this trial, the team redesigned the software to support more asynchronous communication and to include a variety of communication, information, and entertainment areas.
     • Test 2
       • Usability tests
         • After redesigning the software, the team then ran usability tests in the Microsoft usability labs. Seven participants (four male and three female) were tested. Four of these participants had used chat rooms before and three were regular users.
       • What was learned from the usability tests?
         • When running the usability tests, the team collected masses of data that they had to make sense of by systematical analysis. The following discussion offers a snapshot of their findings
   • Looking to the future
     • Future studies were planned to evaluate the effects of the computers and the software in the Fred Hutchinson Center. The focus of these studies will be the social
     • There is always more to learn about the efficacy of a design and how much users enjoy using a product, especially when designing innovative products like HutchWorld for new environments. This study will provide a longer-term view of how HutchWorld is used in its natural environment that is not provided by the other evaluations.

3. Discussion

   • the OMS study illustrated how many different techniques, some of which were highly opportunistic, can be brought into play depending on circumstances. Some practical issues that evaluators routinely have to address include:
     • what to do when there are not many users
     • how to observe users in their natural location (i.e., field studies) without disturbing them
     • having appropriate equipment available
     • dealing with short schedules and low budgets
     • not disturbing users or causing them duress or doing anything unethical
     • collecting "useful" data and being able to analyze it
     • selecting techniques that match the evaluators' expertise

4. Naomi Macias Honti

   • A01282098