Regression

Regression

1. Linear Regression

   • When the relationship between variables x and y can be described with a straight line
   • Correlation - determines strength of relationship between x and y
     • Doesn't tell us how much y changes
   • Regression - estimate how much y will change as a result of a change in x
   • Terminology
     • Variable being predicted
       • y
       • Outcome variable
       • Dependent variable
       • Criterion variable
     • Variable being used to predict
       • x
       • Predictor variable
       • Independent variable
       • Explanatory variable
   • Interpretation
     • Researchers use to
       • Investigate strength of effect
       • Estimate how much y will change
       • Predict a future value of y
     • Makes assumption y is dependent on x
     • Dependence does not reflect causal dependency
     • Regression does not mean causality
   • 3 stages
     • 1) Analyse relationship - correlation
     • 2) Proposing a model to explain relationship - line of best fit
       • a - the intercept
         • value of y when x is 0
       • b - the slope
         • how much y changes as a result of 1 unit increase in x
     • 3) Evaluating model - assessing goodness of fit
       • Simplest Model
         • Assumes no relationship
         • b=0
       • Best Model
         • Based on relationship
         • The regression line
       • Calculating Goodness of Fit
         • SSt - difference between observed values of y and mean
         • SSr - difference between observed values of y and those predicted by line
         • SSt - SSr = SSm
           • Reflects improvement in regression model compared to simplest
           • Larger SSm = bigger improvement in prediction
       • Assessing Goodness of Fit
         • Use an F-test to evaluate improvement due to the model relative to the variance the model does not explain
         • Uses Mean Squares values
           • MSm = SSm/DFm
           • MSr = SSr/DFr
         • F-ratio -provides a measure of how much the model has improved the prediction of y, relative to the level of inaccuracy of the model.
           • F = MSm/MSr
       • Interpreting Goodness of Fit
         • MSm should be large, and MSr should be small - if regression model good at predicting
         • Null hypothesis - regression model and simplest model are equal
         • p expresses probability of finding an improvement when the null hypothesis is true.
         • A significant result suggests that regression model provides a better fit.
   • Regression equation
     • y = bx + a

2. Simple Linear Regression

   • Assumptions
     • Linearity
       • x and y must be linearly related
       • Relationship can be described by a straight line
     • Absence of outliers
       • Regression sensitive to outliers
       • Should remove such data
     • Normality
       • Residuals should be normally distributed
     • Linearity
       • Residuals should have a straight line relationship with outcome
     • Homoscedasticity
       • Variance of residuals should be the same for all predicted scores
     • Independence of residuals
     • Normal P-P Plot of Regression Standardised Resdiual
       • Data points lie in a reasonably straight diagonal line
     • Scatterplot of Regression Standardised Residual
       • Residuals will be roughly rectangularly distributed
       • Outliers - standardised residuals > 3.3 or < -3.3
   • Estimating Variance
     • R^2 - amount of variance explained by the model relative to the total variance
       • R^2 = SSm/SSt
     • Expressed as a percentage by multiplying it by 100
     • r^2 - proportion of shared variance in correlation
     • R^2 is equivalent to r^2 with only 1 predictor
     • Square root of R^2 = r with only 1 predictor

3. Multiple Regression

   • Assess influence of several predictor variables on the outcome variable
   • How much variance in outcome variable the predictor variables combined explain
   • How much variance in the outcome variable predictor variables explain considered separately
   • Assumptions
     • Sufficient sample size
       • Combined effect of several predictors - N >50 + 8M
       • Separate effect of several predictors - N>104 + M
     • Linearity
     • Absence of outliers
     • Multicollinearity
       • Predictor variables should be correlated with outcome variable but not one another
       • Highly correlated predictor variables are measuring the same thing - combine or remove one
     • Normality
     • Homoscedasticity
     • Independence
   • Regression Equation
     • y = b1x1 + b2x2 + a