Diff_Bagging_VS_boosting

Diff_Bagging_VS_boosting

1. Bagging

   • Generate n different bootstrap training sample with replacement
     • Train Algorithm on each bootstrapped sample separately
       • Average the predictions at the end
   • 1)parallel ensemble: each model is built independently 2)aim to decrease variance, not bias 3)suitable for high variance low bias models (complex models) 4)an example of a tree based method is random forest, which develop fully grown trees (note that RF modifies the grown procedure to reduce the correlation between trees)

2. Boosting

   • Draw a random subset of training samples d1 without replacement from the training set D to train a weak learner C1
     • Train new models with higher weights for mis-classified obs
       • Combine all the weak learners via majority voting i.e. weighted average for models giving higher weight which performas better in training data
   • 1)sequential ensemble: try to add new models that do well where previous models lack 2)aim to decrease bias, not variance 3)suitable for low variance high bias models 4)an example of a tree based method is gradient boosting