MacKenzie, 2013, Chapter 5 - Designing HCI Experiments

5.1 What methodology?
1. The way the experiment is designed and carried out
  1. Decide the people (participants), the hardware and software (materials or apparatus), the tasks, the orfer of the tasks, the procedure for briefing and preparing the participants, the variables, the data copllected and analyzed, and so on.
    1. ACM (Association for Computing Machinery), is the dominant organization overseeing computer science and related special interests such as HCI
    2. APA (American Psychological Association), is the dominant organization overseeing experimental psychology.
      1. Publication: Manual of the American Psychological Association, it's a valuable resource for researchers undertaking experiments involving human participants
2. Is the bedrock of science, if the method is weak or flawed, there's no science forthcoming
  1. An ad-hoc or made-up methodology wakens research
5.2 Ethics approval
1. Crucial step that precedes the design of every HCI experiment, due humans are involved.
2. Researchers must respect, safety, welfare, and dignity of human partcipants in their research and them equally and fairly
3. The approval process is governed by the institution or funding agency overseeing the research.
4. Research projects must be accepted by the HRPC (Human Participant Review Community), the IRB (Institutional Review Board), or the ERC (Ethics Review Committee), and so on.
5.3 Experiment design
1. Process of bringing together all the pieces necessary to test hyphoteses on a user interface or interaction technique.
  1. Involves deciding on and defining which variables to use, what tasks and procedure to use, how many participants to use and how to solicit them, and so on.
2. Questions or expressions that help to identify variables
  1. "More quickly" or "Fewer steps" capture the essence of dependent variables.
  2. "With the interface than with an existing interface" capture the essence of an independent variable.
  3. A testable research questions inherently expresses the relationship between an independent and a dependent variable.
5.4 Independent variables
1. Is a circumstance or characteristic that is manipulated or systematically controlled to a change in a human response while the user is interacting with a computer.
  1. Also known as a factor variable.
    1. Experiments designed with independent variables are often called factorial experiments.
      1. Manipulated across multiple (at least two) levels of the circumstance or characteristic.
        Levels are often called test conditions
        A design with a single independent variable includes a main effect
        A design with two independent variables includes two main effects and one interaction effect (two way interaction between variables), for a total of three effects.
        The effect is on the dependent variable
2. Independent because it is independent of participant behaviour, there's nothing a participant can do to influence an independent variable.
3. Typically a nominal-scale attribute, often related to a property of an interface.
  1. In HCI are anything that migth affect users proficiency in using a computer system.
5.5 Dependent variables
1. In HCI is a measured human behaviour
  1. the most common dependent varaibles relate to speed and accuracy
    1. Time often reported as task completion time.
      1. Others include preparation time, action time, thoughput, gaze shifts, mouse-to-keyboard hands transitions, etc
    2. Accuracy often reported as the percentage of trials or other actions performed correctly or incorrectly (successful rates o error rates are used)
  2. Is the variable that depends on the human
2. Are the results or effects of independent variables interactions and manipulations
3. Are used for post-test analysis
5.6 Other variables
1. Control variables
  1. It's a circumstance, facotr or constant that influence a dependent variable, and must remain unchanged throughout the course of the investigation, to influence the outcome.
2. Random variables
  1. Instead of controlling the circumstances, or factors, sometime some migth be allowed to vary ramdomly.
    1. Tipically random variables pertain to characteristics of the participants.
  2. There is a cost since more variability is introduced in the measures, but there is a benefit since results are more generalizable.
    1. Comparisson is best presented in terms of internal and external validity.
3. Confounding variables
  1. Any circumstance or condition that changes systematically with an independent variable.
  2. Are usually problematic variables in research, it is not clear if the effect is presented due the independent variable or to the confounding variable.
    1. Researches must eliminate, adjust or change considerations, to avoid incorrect interpretations in effects.
  3. Influences independent and dependent variables.
4. Receive considerable less attention and are rarely mentioned in research papers, nevertheless are very important in research.
5.7 Task and procedure
1. There are 2 objectives in designing a good task (descriminate test conditions), represent and discriminate.
  1. A task similar to actual or expected usage will improve the external validity (ability to generalize results in other people and situations)
  2. A task that is representative of actual usage may decrease internal validity (the effects observed, i.e. the differences in means on a dependent variable)
2. Task may require a performance-base or skill bases in HCI
3. The experimental procedure includes the task, the instructions, demostration, or practice given to the participants.
5.8 Participants
1. HCI often required people with specific skills, filtering processes might be required to ensureonly appropiate participants are used in the research.
2. Generalize results to people that were not participants, apply when these people are member of the same population to whom results are assumed to hold.
3. How many participants are good enough?
  1. Tip: always investigate similar researches to stablish a base line 'a priori'
  2. Sometimes statistical significance is otained wihout need too much participants.
  3. A large number of participants: statistically significant results for a difference of no practical significance
  4. Not enough participants may prevent statistical significance from appearing
4. Selection method it's also important, sometimes to have better external validity, is required to select parcipants randomly
5. The selection criteria should be clear and stated in the write-up of the methodology, in a section labeled "Participants"
5.9 Questionnaire design
1. Purposes in most HCI experiments
  1. Gather information on demographics (age, gender, etc)
  2. Experience with related technology
  3. Are also used at the end of the research to obtain opinions and feelings from participants about the interfaces or interaction techniques
2. Questionnaires are the primary instrument for survey research
  1. Sometimes when giving response options, is useful to add the option 'Other'
  2. Have in mind the question be useful to easily calculate standard deviations, relationships in data for ratio-scale, include ratio-scale items helps to find correlated responses
    1. Ordinal responses helps to obtain ordinal data, although inherently is lower data quality than ratio-scale
5.10 Within-subjects and between-subjects
1. Within-subjects
  1. Each participant is tested on each level.
  2. Also called repeated measures, because the measurements on each test condition are repeated for each participant.
  3. Requires fewer participants but more testing for each participant
  4. The variance due to paticipants will be approximately the same across test conditions
  5. It is not necessary to balance groups, because there's only one group
2. Between-subjects
  1. Each participant is tested only in one level.
  2. Balance groups of participants is required to ensure the groups are more or less equal in terms of characteristics that might introduce bias in the measurements
    1. Balancing is usually done through a random assigment, but also assignation according to reasonable criteria
  3. Avoid interference (conflict that arises when a participant is exposed to one test condition and then switches to another test condition) between test conditions
    1. The majority of facotrs that appear in HCI experiments are like this
  4. More participants are needed to obtain the same number of observations for each test condition
3. How to administering test conditions (levels or factors) in participants
4. For an experiment with two factors it is posible top assign the levels of one factor within-subjects and the levels of the other factor between-subjects.
  1. This is known as mixed design
5.11 Order effects, counterbalancing, and latin squares
1. Order effects occurs depending on the nature of the task
  1. Practice in within-subjects is a confounding variable due the practice increases systematically from one test condition to the next.
    1. Known as practice effect or learnig effect
      1. An effect of fatigue may follow after this, and so on.
    2. When the experiment has confounding influence, it's necessary make a compensatory ordering of conditions
      1. Most common method of compensating order effects is to divide participants into groups and administer conditions in different order
2. Counterbalancing
3. A lating square is an n X n table filled with n different symbols positoned such that each symbol occurs exactly once in each row and each column.
5.12 Group effects and asymmetric skill transfer
1. There are cases where different effects appear for one order A -> B compared to another B -> A
  1. In such cases there migth be a group effect (differences across groups in the mean scores on a dependent variable)
    1. Counterbalancing does not work
    2. Usually due to asymmetric skill transfer (differences in the amount of impromented, depedning on the order of testing)
5.13 Longitudinal studies
1. An experimental evaluation where participants practice over a prolonged period.
  1. Sometimes the research has a particular interest in learning, or the acquisition of skill.
    1. In this case the experimental procedure invovles testing user over a prolonged period while their impromevement in performance is measured.
      1. Learning is not eliminated using methods as counterbalancing, order effects or learning effects are important in this case.
2. Crossover point and cost-benefit
5.14 Running the experiment
1. Experment design ✓, Apparatis build and tested ✓, Participant recruited and scheduled ✓
  1. Are you sure the time to begin the experiment has arrived?
2. It is always useful to have a pilot test
  1. It can be with oen or two participants
    1. Helps to:
      1. Smooth out the prtocol for briefing
      2. Preparint participants
      3. Check the amount of time per participant
      4. Check if instructions are clear enough for participants
      5. Avoid undepressure on participants
      6. Avoid conveying indifference or disregard

Next up

MacKenzie, 2013, Chapter 5 - Designing HCI Experiments

Description

Resource summary

Media attachments

Similar

	Created by OSCAR ALEJANDRO ROMERO RODRIGUEZ almost 4 years ago