Evaluation

Evaluation

1. Evaluation setup

   • We would want our model to perform well on future data, without training​ It’s not very useful to be able to predict instance(i) is class(i)​ It’s easy to be perfect on training data​ Does not mean you will do well on future data
   • Generalization
     • How do we minimize future error?​ Avoiding over-fitting and under-fittin
     • Over-fitting​
       • When our model can predict the training data perfectly (with few mistakes) but does not perform well on unseen future data​
       • The model thinks that all the data (including noise data) is important and correct​
       • too complex (too specific)
       • low bias (approximation error) but high variance​
     • Under-fitting
       • When our model cannot capture the underlying trend of the data Training is def. bad, but not sure about test​
       • The generated model is too simple (too general)
       • shows low variance but high bias (approximation error)​
   • Estimating Error​
     • To know whether our learnt model under-fits, over-fits or is just right​
     • Test data​
       • Set aside part of the training data Ensure that each class is represented with approximately equal proportions in both training and test data​
     • Train
       • Train our model without using any of the test data​
     • Calculate
       • Calculate our models performance when tested on the test data​
       • If the performance of our model does not decline when we test it on the test data​ The model is just right​ Generally, simple models outperform complex models​
     • Bias
       • Cross Validation
   • Parameter Tuning
     • Each dataset needs different levels of flexibility​

2. Confidence Interval

3. Standard Deviation

4. Paired T Test

Images and files in the map