P-values, Generalisability, Study Limits

P-values, Generalisability, Study Limits

1. Summary

   • tests of significance
     • allow the researchers to formally test their null hypothesis and to demonstrate the strength of that evidence in any publication.
     • In many cases, a research finding is not always accepted from one published article based on a single study.
   • many other researchers will retest the null hypothesis
     • using perhaps stronger or more powerful studies in order to improve the level of evidence
       • until there are a number of published articles from various types of studies before clinicians will change their practices and adopt the findings from the published research. This
   • important that researchers published fully transparent results from their research.
     • Subsequent researchers will re-test the null hypothesis using the earlier studies to improve their own research into the same health problem.

2. Generalisability

   • is the extent to which research findings and conclusions from a study conducted on a sample population can be applied to the population at large.
     • The first step to check generalisability is to ensure that the result was statistically significant
   • aka The first step to check generalisability is to ensure that the result was statistically significant
   • Study validity
     • the degree to which an inference from a scientific study is warranted,
     • taking into the account the strengths and weaknesses of the study design.
   • Internal validity
     • the degree to which observations taken during the study may be attributed solely to the hypothesized effect that is being studied
   • External validity
     • is the extent to which the finding of a study can be generalised from the results from the sample being studied, to the population from which the sample was taken.

3. Important note

   • The results of any study may not be generalisable to all patients outside the study setting
     • example, when researchers conduct an RCT, they usually address the question of whether a treatment can work (efficacy) but may not tell us about whether the treatment will be effective when offered to the broad range of patients seen in day-to-day clinical practice.

4. Study Limitations

   • The researchers should always point out the shortcomings of the study, especially those that affect the conclusions drawn
     • then "defend" the results and conclusions drawn in light of these limitations.
       • You need to assess whether or not these limitations damage the overall nature of study findings?
         • limitations
           • might relate to the overall study design, the response rate, the characteristics of the sample etc.
             • The limitations might affect the degree to which the results can really be generalised to the target populations from which the study sample was drawn.