Probability, Effect Sizes, and Repeated Measures Designs (Part 1)

Probability, Effect Sizes, and Repeated Measures Designs (Part 1)

1. Probability

   • Returning to our example: How do we decide when an observed difference is large enough to allow us to reject the null hypothesis, i.e. is the gender difference in Classwork 1 means (3.33) big enough for us to reject the null hypothesis?
     • The research community has decided that a useful rule of thumb to apply here is whether or not the difference would occur by chance only 1 in 20 = 0.05 or a 5% probability) if the null were true.
       • This corresponds to the 5% of difference scores which are extreme (the biggest differences)
         • Recall that 95% of scores fall within +/- 1.96 z-scores. So if we took lots of samples of men and women, and kept plotting the difference between the two, we would see the following graph. Plot of the distribution of differences between men's and women's Classwork 1 scores if we took lots and lots of repeated samples of men and women.
           • So, when a p-value (probability) is higher than 5% (reported as p > .05) it indicates that the difference you have observed is quite likely to have occurred if the null hypothesis were true.
             • In this case, you would accept the null hypothesis and conclude that there is no effect of your IV on your DV
               • so - if p > .05 then you would not reject the null hypothesis. This is because the difference between your two group means fall in the green region (within the 95% of likely random variation around the '0' that represents no difference). You've discovered that the difference you have observed is one which would occur quite frequently purely by chance.

Images and files in the map