Machine Learning Strategies

Machine Learning Strategies

1. Orthogonality of Tuning Parameters

   • Like a TV screen, try using tuning parameters that directly affect the issue (e.g. affect only trapezoidicity of tv screen), and does not make the side-effect. This is the orthogonality, using tuning parameters which are orthogonal to each other

2. Single Number evaluation metric

   • Based on multiple performance measuring metrics, try to build one value formula, which would help you pick easier to right option.

3. Human-level Performance

   • In Machine Learning, algorithm accuracy grows over time, might pass above the human level performance, but will saturate over limit that is called Bayes optimal error
   • Avoidable bias
     • Is the difference between between human level performance and training set error. if you assume that human level performance is almost as good as bayes optimal error, it defines a limit to what accuracy machine learning can achieve.
     • Can be corrected with: (1) Train bigger model, (2) Train longer/better optimization algorithms (momentum, RMSprop, adam), (3) better NN architecture / hyperparameter search

4. Reducing (avoidable) bias and variance

   • Variance
     • Can be corrected with: (1) More Data, (2) Regularization (L2, dropout, data augmentation), (3) NN Architecture / hyperparameter search

5. Error Analysis

   • 1) "ceiling" look on the problem
     • Should you take another classifier (train for picture of dogs): Take around ~100 mislabeled samples, and count manually how many are actually mislabeled as another class (dogs),
       • So if it is low percentage of such mislabeling, your effort to add another classifier would not reduce error
       • If it is high percentage, it is worth the effort, as it would reduce the error
   • 2) Evaluate multiple ideas in parallel
     • Create a table with all ideas, and cross compare the percentage of how much each idea is appearing in dataset of mislabeled samples, and thus how much would they affect your performance
   • 3) Correcting incorrect dev/test sets (incorrectly labeled)
   • 4) Build quickly, and then iterate
   • Mismatched distributions of training set and test set
     • Issue here is: if your train error is 1% but test error is 10%, you cant be sure if it is high variance problem of data mismatch problem.
       • Addressing data mismatch problem:
         • 1) carry out error analysis (e.g. car noise, backgroudn music...)
         • 2) Make training data more similar to dev/test set
     • Can be fixed by introducing training/dev set which is of the same distribution like training set

6. When Transfer learning from Task A to Task B makes sense?

   • When Task A and Task B have the same input
   • You have a lot more data for Task A than Task B
   • Low level features of task A could be helpful for Task B

7. When multitask learning makes sense?

   • a set of tasks can benefit from low level features of NN
   • amount of input data for each task is similar

8. end-to-end deep learning

   • when u have a pipeline of tasks to learn - deep NNs perform better, only when there is a large input training data available.
   • Pro: Let the data speak (not inventing something and distract NN, e.g. phonemes)
   • Pro: Less hand-designing components needed
   • con: May need large amount of data
   • con: Excludes potentially useful hand-designed components

Bilder und Dateien der Mindmap