Flashcards
Effect Size and Intervals and Confidence Intervals
Cards in this deck (16)
- 1We focus a lot on p-values…• So it’s easy to forget that a p-values is not a
measure of the strength of an effect
• A p-value is the chance you would have
observed what you did if the null hypothesis
(no effect) is true
• In other words, it’s about how strongly we
believe in an effect
• This is not the same as an estimate of the
strength of the effect - 2Effect size is the measure of the strength of the effectYou are already familiar with one effect size…
– The correlation coefficient r
– Remember, this is different to p - 3Effect size is the measure of the strength of the effectWhat about two-sample comparisons?
– E.g. comparing a treatment group with a control
group that didn’t get treatment.
– Let’s say we have a new treatment for
triskaidekaphobia (fear of the number 13).
• You would likely do a two-sample t-test to see
if we believe there is an effect.
• To accompany this, we could use Cohen’s d to
see how big this effect is. - 4Strenght of effect and two meansThe strength of
the effect is not
necessarily the
same as the
difference
between the two
means. It also
depends on the
variation.
More variation =
more overlap =
less effect - 5Cohen’s d for a two-sample comparison• Cohen’s d expresses the mean difference
between two groups in standard deviation
units
• d = ( M2
- M1
) / SD
• (Each group might have a different SD – the full Cohen’s d
formula is a bit more complicated because it works out the
average SD for the two groups).
• So think of Cohen’s d like this: How many
standard deviations apart are the means? - 6Cohen’s d for paired-sample (or one sample) comparisons• For these types of comparison, for each
participant we focus on a difference value:
– For paired-samples, it’s the difference between the
two values for each participant. E.g. difference in
phobia level before and after treatment.
– For one-sample, it’s the difference between the data
point and whatever value is being compared against
• d = MDifference / SDDifference
• So think of Cohen’s d like this: How many
standard deviations is the difference - 7Interpreting Cohen’s d• What counts as a big or small effect is to some
extent subjective, but there are some
standard rules of thumb in psychology for
interpreting the size of d.
– d = .2 means a small effect
– d = .5 means a medium effect
– d = .8 means a large effect - 8NoiseYou might say these all look
like pretty small effects. The
thing is, in psychology, there
is almost always so much
“noise” (unexplained
variance) that effects don’t
tend to get much bigger –
there will usually be a lot of
overlap between groups
because of noise.
Small effects won’t even be
detected unless you have a
pretty large sample size.
How large is “large”? This is
an issue of power, outside
the scope of the module. - 9Effect size for ANOVA: Partial eta-squared (η2p)• Remember ANOVA is about explaining variance.
• So for ANOVA, the standard effect-size η
2
p
is a
measure of how much variance in the DV is
explained by the IV.
• There is a η
2
p
for each IV.
• SPSS will output η
2
p when you use the General
Linear Model function.
• η
2
p is particularly useful for IVs with more that
two categories (otherwise you could just use d)
• Again, there are somewhat arbitrary but rather
standard rules of thumb for interpretation:
– Small = .01; Medium = .06; Large = .14 - 10Confidence intervals• Remember, sample descriptions like the mean or
the standard deviation (or even the effect size)
are usually only interesting because they are
estimates of the true value in the population.
• There is uncertainty attached to this estimate.
• We can use confidence intervals to quantify this
uncertainty.
• E.g. if we have a sample mean of 9, we might also
have a relevant 95% confidence interval of 7.5 to
10.5.
• What this (roughly) means is that there is a 95%
probability that the true population mean is in
this interval - 11Confidence intervals are often displayed graphicallyThe sample mean is 9, and
we are 95% confident that
the population mean is
between 7.5 and 10.5
• (Often error bars like this
will be standard error, but
95% CI is better)
• The bigger the sample, the
more confident we will get
in our estimates, so the
confidence interval will get
narrower - 12You can put a confidence interval on pretty much any estimate• (But we are currently just interested in means,
and effect size, specifically, Cohen’s d).
• p-values have a lot of associated problems, that
modern statisticians argue a lot about.
Confidence intervals on effect sizes can actually
be used as a replacement for p-values, and this
solves some of these problems.
• The basic idea is that if an effect size of 0 is
outside the confidence interval, then we are
confident that there is an effect, otherwise we
are not. - 13Thinking in terms of CI on effect size helps understanding a lot• Often students (and those who should know
better) will write things like “there was a
significant effect of A compared to control
condition, but no significant effect of B.
Therefore A has more effect than B”.
• If you think carefully about what significance
actually means, you should realise this is
nonsense (see next slide). - 14Easy was to remember confidence intervals• If the 95% confidence interval for the effect
size excludes 0, then p < .05.
• If the 95% confidence interval for the effect
size includes 0, then p > .05.
• So p is redundant if you have an effect size
confidence interval. This intervals tells you
everything p tells you, and more.
• We keep on reporting p, to some extent just
because of convention - 15The relation between n, p, and effect size• As long as there really is even just the tiniest
effect (and there usually is) then
– As n goes up, p will go down
– As n goes up, the effect size won’t change
systematically, but our confidence in the estimate
will increase
• As the effect size goes up, p will tend to
become more significant, because larger
effects are easier to detect - 16Effect size is about practical significance, as opposed to statistical significance• Say you develop a new treatment and test its
effectiveness. You have the resources for a really
large-scale trial (large n). Maybe you become
really sure that the treatment has an effect (small
p), but you are also really confident (due to the
large n) that the effect size is small, e.g. 95% CI
for Cohen’s d [.15, .19].
• Should the NHS fund this treatment? Maybe not:
– We are sure it does something, but we are sure that
this something isn’t much.
• (By the way, that square brackets thing is APA
format for reporting confidence intervals.)
Information
- Description
- U24137: Advanced Statistics and Experimental Method for Psychology Flashcards on Effect Size and Intervals and Confidence Intervals, created by Jasmine Stow on 27/11/2018.
- Created by
- Jasmine Stow
- Views
- 1
- Rating
More on Point Estimation and Confidence Intervals