Measuring the User Experience (Tullis and Albert:2013)

4. Performance metrics
1. Task success
  1. Binary success
  2. Levels of success
  3. Issues in measuring success
2. Time on task
  1. Importance
  2. Collecting and measuring
  3. Analyzing and presenting
  4. Issues to consider
3. Errors
  1. When to measure
  2. What is
  3. Collecting and measuring
  4. Analyzing and presenting
  5. Issues to consider
4. Efficiency
  1. Collecting and measuring
  2. Analyzing and presenting
  3. As a combination of task success and time
5. Learnability
  1. Collecting and measuring
  2. Analyzing and presenting
  3. Issues to consider
5. Issue-based metrics
1. What is an issue?
  1. Real vs. false
2. How to identify
  1. In-person study
  2. Automated study
3. Analyzing and reporting metrics for usability issues
  1. Freq of unique issues
  2. Freq of issues per participant
  3. Freq of participants
  4. Issues by category
  5. Issues by task
4. Consistency and bias
5. Number of participants
  1. 83% of issues by 1st 5 people (31% avg prob of any one issue)
    1. Other study only 35% of issues
      1. Tullis: Catch big issues; Homogenous user group; Limited design
        More people when all issues must be caught; various user groups; Large design
6. Severity ratings
  1. Based on UX
  2. Based on combination of factors
  3. Using rating system
    1. Caveats
6. Self-reported metrics
1. Importance: User perception; Insight into feelings
2. Rating scales
  1. Likert
  2. Semantic differential scales
  3. How and when:
  4. Analyzing
3. Post-task rating
  1. Ease of use
  2. After-Scenario Questionnaire (ASQ)
  3. Expectation measure
4. Post-session ratings
  1. Aggregating individual task ratings
  2. System Usability Scale
  3. Computer System Usability Questionnaire
  4. Questionnaire for UI satisfaction
  5. Usefulness, Satisfaction, and Ease-of-Use Questionnaire (USE)
  6. Product reaction cards
5. Other types
  1. Specific attributes
  2. Specific elements
  3. Open-ended questions
  4. Awareness and comprehension
  5. Awareness and usefulness gaps
7. Behavioural and Physiological Metrics
1. Unprompted verbal expressions
2. Eye tracking
  1. Scan path
  2. Heat map
  3. Areas of Interest (AOIs)
  4. Binning chart
  5. Metrics
    1. Dwell time
      1. # of fixations
        Fixation duration
        Sequence
        Time to First Fixation
        Revisits
        Hit ratio
  6. Analysis
  7. Pupillary response
3. Emotion
  1. Affectiva; Q-Sensor; Blue Bubble Lab; Emovision; Seren; Emotiv
4. Stress and physiological measures
  1. Heart rate variance
  2. Skin conductance
5. PressureMouse; Posture Analysis Seat
8. Combined and comparative metrics
1. Single Usability Scores
  1. Based on target goals
  2. Based on Percentages
  3. Based on Z scores
  4. Using single usability metric
2. Usability scorecards
3. Comparisons against
  1. Goals
  2. Expert performance
The basics
1. 1. Introduction
  1. Definitions
    1. UX: User involved, interact with interface, measurable and interesting
  2. Value of UX metrics
    1. Structure, extent, evaluation of changes, ROI, patterms
  3. Ten myths
    Annotations:
    - 1. Take too much time 2. Too expensive 3. Not useful for small improvements 4. Don't help understand causes 5. Too noisy 6. Trust your gut 7. Don't apply to new products 8. No metrics exist for this 9. Management doesn't care 10. Can't collect data with small sample
    1. 1. Take too much time
      1. 2. Too expensive
        3. Not useful for small improvements
        4. Don't help understand causes
        5. Too noisy
        6. Trust your gut
        7. Don't apply to new products
        8. No metrics exist for this
        9. Management doesn't care
        10. Can't collect data with small sample
2. 2. Background / Statistics
  1. Independent and dependent variables
    1. Dependent: Things you measure (e.g. # of errors)
    2. Independent: Things you control (e.g. Age, designs)
  2. Types of data
    1. Nominal
    2. Ordinal
    3. Interval
    4. Ratio
  3. Descriptive statistics
    1. Central tendency
    2. Variability
    3. Confidence intervals
  4. Comparing means
    1. Independent samples
    2. Paired samples
    3. >2 samples
  5. Relationships between variables
    1. Correlations
  6. Nonparametrics tests
    1. Chi square (X2)
  7. Graphical presentation
    1. Column or bar chart
      1. Line graphs
        Scatterplots
        Pie or donut chart
        Stacked bar or column
3. 3. Planning
  1. Study goals
    1. Formative study
    2. Summative study
  2. User goals
    1. Performance
    2. Satisfaction
  3. Choosing the right metrics
    1. TEN TYPES OF USABILITY STUDIES
  4. Evaluation methods
    1. Traditional
    2. Online
    3. Online surveys
  5. Other study details
    1. Budget and timeline
    2. Partcipants
    3. Data collection
    4. Data cleanup

Next up

Measuring the User Experience (Tullis and Albert:2013)

Description

Resource summary

Media attachments

Similar

	Created by William Dunn over 5 years ago