An evaluation framework

An evaluation framework

1. Evaluation paradigms and techniques

   • Evaluation paradigms
     • "Quick and dirty" evaluation
       • is a common practice in which designers informally get feedback from users or consultants to confirm that their ideas are in line with users' needs and are liked.
       • it is meant to be done in a short space of time.
     • Usability testing
       • involves measuring typical users' performance on carefully prepared tasks that are typical of those for which the sys- tem was designed. Users' performance is generally measured in terms of number of errors and time to complete the task.
       • This observational data is used to calculate performance times, identify errors, and help explain why the users did what they did.
     • Field studies
       • In product design, field studies can be used to (1) help identify opportunities for new technology; (2) determine requirements for design; (3) facilitate the introduction of technology; and (4) evaluate technology
       • In some field studies the evaluator may be an insider or even a participant. Ethnography is a particular type of insider evaluation in which the aim is to explore the details of what happens in a particular social setting.
     • Predictive evaluation
       • experts apply their knowledge of typical users, often guided by heuristics, to predict usability problems. Another approach involves based models. The key feature of predictive evaluation is that users need not be pres- ent, which makes the process quick, relatively inexpensive, and thus attractive to companies; but it has limitations.
   • Techniques
     • Observing users
       • help to identify needs leading to new types of products and help to evaluate prototypes.
     • Asking users
       • Interviews and questionnaires are the main techniques for doing this. The questions asked can be unstructured or tightly structured.
     • Asking experts
       • Software inspections and reviews are long established techniques for evaluating software code and structure
     • User testing
       • Measuring user performance to compare two or more designs has been the bedrock of usability testing.
     • Modeling users' task performance
       • There have been various attempts to model human-computer interaction so as to predict the efficiency and problems associated with different designs at an early stage without building elaborate prototypes.

2. DECIDE: A framework to guide evaluation

   • Determine the overall goals that the evaluation addresses
     • Check that the evaluators have understood the users' needs.
     • Identify the metaphor on which to base the design.
     • Check to ensure that the final interface is consistent.
     • Investigate the degree to which technology influences working practices.
     • Identify how the interface of an existing product could be engineered to improve its usability.
   • Explore the specific questions to be answered.
     • In order to make goals operational, questions that must be answered to satisfy them have to be identified.
       • Questions can be broken down into very specific sub-questions to make the evaluation even more specific.
   • Choose the evaluation paradigm and techniques to answer the questions.
     • Having identified the goals and main questions, the next step is to choose the evaluation paradigm and techniques.
       • what seems to be the most appropriate set of techniques may be too expensive, or may take too long, or may require equipment or expertise that is not available, so compromises are needed.
   • Identify the practical issues that must be addressed, such as selecting participants.
     • There are many practical issues to consider when doing any kind of evaluation and it is important to identify them before starting. Some issues that should be considered include users, facilities and equipment, schedules and budgets, and evaluators' expertise.
       • Users
         • It goes without saying that a key aspect of an evaluation is involving appropriate users. For laboratory studies, users must be found and screened to ensure that they represent the user population to which the product is targeted.
       • Facilities and equipment
         • There are many practical issues concerned with using equipment in an evaluation. For example, when using video you need to think about how you will do the recording: how many cameras and where do you put them?
       • Schedule and budget constraints
         • Time and budget constraints are important considerations to keep in mind. It might seem ideal to have 20 users test your interface, but if you need to pay them, then it could get costly.
       • Expertise
         • Does the evaluation team have the expertise needed to do the evaluation? For example, if no one has used models to evaluate systems before, then basing an evaluation on this approach is not sensible. It is no use planning to use experts to review an interface if none are available.
   • Decide how to deal with the ethical issues.
     • The Association for Computing Machinery (ACM) and many other professional organizations provide ethical codes that they expect their members to uphold, particularly if their activities involve other human beings. For example, people's privacy should be protected, which means that their name should not be associated with data collected about them or disclosed in written reports
   • Evaluate, interpret, and present the data.
     • Choosing the evaluation paradigm and techniques to answer the questions that satisfy the evaluation goal is an important step. So is identifying the practical and ethical issues to be resolved. However, decisions are also needed about what data to collect, how to analyze it, and how to present the findings to the development team.
     • Reliability
       • is how well it produces the same results on separate occasions under the same circumstances.
     • Validity
       • is concerned with whether the evaluation technique measures what it is supposed to measure.
     • Biases
       • occurs when the results are distorted. For example, expert evaluators performing a heuristic evaluation may be much more sensitive to certain kinds of design flaws than others.
     • Scope
       • refers to how much its findings can be generalized. For example, some modeling techniques, like the keystroke model, have a narrow, precise scope.
     • Ecological validity
       • concerns how the environment in which an evaluation is conducted influences or even distorts the results.

3. Pilot studies

   • It is always worth testing plans for an evaluation by doing a pilot study before launching into the main study. A pilot study is a small trial run of the main study.
   • A pilot study will identify potential problems in advance so that they can be corrected. Sending out 500 questionnaires and then being told that two of the questions were very confusing wastes time, annoys participants, and is expensive.

4. Naomi Macias Honti

   • A01282098

Images and files in the map