Evaluating Designs

Evaluating Designs

1. Evaluate empirically

   • Usability tests
     • Target/representative users test
       • Monitor user's path
     • Find lower level problems
       • Breakdowns
     • Define tasks and goals
       • Define "correct" path
     • Retrospective interview
       • User reflect on recording
   • Tech probes
     • Put prototype in context
       • Record how prototype changes practices
   • Experience sampling
     • Interrupt with surveys
     • Ecological validity
   • A/B Tests
     • Compare designs experimentally
       • Current control vs new experimental
     • Difficult to measure success sometimes

2. How to be Critical

   • Critique
     • Collaborative, constructive feedback
     • Two-way
       • Counterarguments
     • Engage deeply
     • Divergent/convergent
     • Forms of Critique
       • Compliment sandwich
       • Socratic questioning
         • Clarify design
         • Challenge designer
         • Think of implications
   • Domain expertise
     • Critique session with stakeholders
   • Principles of Good designs
     • Vague
       • Intuitive
       • User-friendly
     • Precise (not universally good)
       • Simple
       • Novel
       • Powerful
       • Invisible
       • Universal
       • Just

3. Analytical evalutation

   • Simulate people using a design
     • Heuristic evaluation
       • Is interface learnable?
       • 4-5 experts inspect each aspect
         • Heuristcs
           • System status visible
           • Error prevention
           • Real world match
           • User control
           • Consistency
           • Recognition v Recall
           • Flexibility
           • Recover from errors
       • Exhausting process so learn to apply as they design
     • Walkthroughs
       • Expert completes tasks step by step
         • Select task, use prototype, develop persona
           • Will user achieve the right steps and do correct actions? Will they see they are correct/wrong?
           • GenderMAG -covers broad spectrum
             • Custom personas - motivations, self-efficacy, etc.
       • Systematic & granular
     • Claims analysis
       • how a design supports or doesn't support a claim
         • identifies goals
     • Cognitive modeling
       • Build GOMs
         • optimize expert behavior w/o user testing