An Evaluation Framework

An Evaluation Framework

1. Designing useful and attractive products requires skill and creativity.

   • But how do evaluators decide what and when to evaluate?
     • Identifying usability and user experience goals is essential for making every product successful, and this requires understanding users' needs.
       • As designers under- stand users' needs better, their designs reflect .this understanding.
         • Evaluation is driven by questions about how well the design or particular as- pects of it satisfy users' needs.
           • Planning evaluation studies involves thinking about key issues and asking questions about the process

2. Evaluation paradigms and techniques

   • User Studies: Essentially involve looking at how people behave either in their natural [environments], or in the laboratory, both with old technologies and with new ones."
     • Any kind of evaluation, whether it is a user study or not, is guided either explicitly or implicitly by a set of beliefs that may also be un- derpinned by theory. These beliefs and the practices, are known as an evaluation paradigm,
     • The techniques associated with usability testing are
       • "Quick and dirty" evaluation
         • Common practice in which designers informally get feedback from users or consultants to confirm that their ideas are in line with users' needs and are liked. because it is meant to be done in a short space of time. Getting this kind of feedback is an essential ingredient of successful design. fed back into the design process
         • Usability testing
           • Usability testing involves measuring typical users' performance on carefully prepared tasks that are typical of those for which the sys- tem was designed. Users' performance is generally measured in terms of number of errors and time to complete the task
             • It is strongly controlled by the evaluator. There is no mistaking that the evaluator is in charge!
           • Field studies
             • field studies is that they are done in natural settings with the aim of increasing understanding about what users do naturally and how technology impacts them. interviews, observation, participant observation, and ethnography that are used in field studies
               • The first involves observing explicitly and recording what is happening, as an outsider looking on. Qualitative techniques are used to collect the data,
               • The evaluator may be an insider or even a participant. Ethnography is a particular type of insider evaluation in which the aim is to explore the details of what happens in a particular social setting.
             • Predictive evaluation
               • Experts apply their knowledge of typical users, often guided by heuristics,to predict usability problems. Key feature of predictive evaluation is that users need not be pres- ent, which makes the process quick, relatively inexpensive, and thus attractive to companies; but it has limitations.

3. Techniques

   • Observing users: Observation techniques help to identify needs leading to new types of products and help to evaluate prototypes. Notes, audio, video, and interaction logs are well- known ways of recording observations and each has benefits and drawbacks.
     • Asking users what they think of a product- whether it does what they want; whether they like it; whether the aesthetic design appeals; whether they had problems using it; whether they want to use it again- is an obvious way of getting feedback.
       • Asking experts Software inspections and reviews are long established techniques for evaluating software code and structure. Guided by heuristics, experts step through tasks role-playing typical users and identify problems.
         • User testing: Measuring user performance to compare two or more designs has been the bedrock of usabilitytesting
           • Modeling users' task performance: There have been various attempts to model human-computer interaction so as to predict the efficiency and problems associated with different designs at an early stage without building elaborate prototypes.

4. DECIDE

   • 1) Determine the overall goals that the evaluation addresses.
     • What are the high-level goals of the evaluation? Who wants it and why? An evalua- tion to help clarify user needs has different goals from an evaluation to determine the best metaphor for a conceptual design, or to fine-tune an interface
     • Explore the questions
       • Questions that must be answered to satisfy them have to be identified. Questions can be broken down into very specific sub-questions to make the evaluation even more specific
       • Choose the evaluation paradigm and techniques
         • The evaluation paradigm determines the kinds of techniques that are used.
         • Identify the practical issues
           • when doing any kind of evaluation and it is important to identify them before starting. Some issues that should be consid- ered include users, facilities and equipment, schedules and budgets, and evaluators
           • Users
             • key aspect of an evaluation is involving appropriate users. Another issue to consider is how the users will be involved. no written rules about the length of time that a user should be expected to spend on an evaluation task. Greeting users, explaining that it is the system that is being tested and not them,
             • Facilities and equipment
               • There are many practical issues concerned with using equipment in an evaluation. For example, when using video you need to think about how you will do the recording: how many cameras and where do you put them?
               • Schedule and budget constraints
                 • Time and budget constraints are important considerations to keep in mind. It might seem ideal to have 20 users test your interface, but if you need to pay them, then it could get costly. Planning evaluations that can be completed on schedule is also im- portant, particularly in commercial settings.
                 • Expertise
                   • Does the evaluation team have the expertise needed to do the evaluation? For ex- ample, if no one has used models to evaluate systems before, then basing an eval- uation on this approach is not sensible.

5. ARANZA ESTRADA A01176250

   • Preece, Jennifer. (2002). Interaction Design. United States of America: John Wiley & Sons Inc.

6. Ethical issues

   • For example, people's privacy should be protected, which means that their name should not be as- sociated with data collected about them or disclosed in written reports (unless they give permission).
     • For example, if a focus group involves nine men and one woman, the pronoun "she7' should not be used in the report because it will be obvious to whom it refers.
       • People give their time and their trust when they agree to participate in an evalua- tion study and both should be respected
         • Tell participants the goals of the study and exactly what they should expect if they participate.
           • Be sure to explain that demographic, financial, health, or other sensitive in- formation that users disclose or is discovered from the tests is confidential.
             • Make sure users know that they are free to stop the evaluation at any time if they feel uncomfortable with the procedure

Images and files in the map