5 -2. Build model pipelines(5. Model Training)

Page 1

2.1 - Preprocessing

Sometimes we'll want to preprocess the training data even more before feeding it into our algorithms.

For example, we may want to...

Transform or scale our features.

Perform automatic feature reduction (e.g. PCA)

Remove correlated features.

The key is that these types of preprocessing steps should be performed inside the cross-validation loop.

Page 2

2.2 - Standardization

Standardization is the most popular preprocessing technique in machine learning.

Standardization transforms all of your features(columns) to the same scale by subtracting means and dividing by standard deviations.

This is a useful preprocessing step because some machine learning algorithms will "overemphasize" features that are on larger scales. Standardizing a feature is pretty straightforward:

First, subtract its mean from each of its observations.

Then, divide each of its observations by its standard deviation.

This makes the feature's distribution centered around zero, with unit variance. To standardize the entire dataset, you'd simply do this for each feature.

Page 3

2.4 - Preprocessing parameters

Now, the manual standardization we just performed is nice for seeing how it works, but it's not how we'll do it in practice.

As mentioned earlier, we need to perform preprocessing steps inside the cross-validation loop.

Here's why:

To perform standardization, we needed 2 pieces of information directly from the training set:

The means of each feature

The standard deviations of each feature

These are called preprocessing parameters because they actually needed to be learned from the training data.

So what if we want to preprocess new data (such as our test set) in the exact same way?

Page 4

2.5 - Standardizing the test set

Well, we'd need to use the exact same preprocessing parameters that we learned from the training set(not those from the test set)!

Mathematically speaking, this preserves the exact original transformation.

Practically speaking, if new observations come in 1 at a time, you can't standardize them without already having preprocessing parameters to use (i.e. those learned from your training set).

You'd subtract the training set's means from the test set's features.

Then you divide the test set's features by the training set's standard deviations.

That's why you actually need to save the preprocessing parameters from the training set!

Page 5

2.6 - Pipelines and cross-validation

So what is a model "pipeline?"

As you'll recall, we'll be tuning (i.e. finding the best settings for) our model with cross-validation.

We'll split the training set into 10 different folds.

Therefore, when we apply preprocessing, we should only learn preprocessing parameters from the 9 training folds in each iteration.

In other words, we need to store our preprocessing steps in a "model pipeline" and perform cross-validation on that entire pipeline.

10-fold cross-validation with preprocessing

Here's how the new 10-fold cross-validation process would look (the new steps are bolded):

Split your data into 10 equal folds.

Preprocess 9 training folds, learning the preprocessing parameters.

Train your model on those newly preprocessed 9 folds.

Preprocess the hold-out fold using the same preprocessing parameters from the training folds.

Evaluate your model on the newly preprocessed hold-out fold.

Perform steps (2) - (5) 10 times, each time holding out a different fold.

Average the performance across all 10 folds.

Page 6

2.7 - How to make a pipeline

Now, what you just learned is a sophisticated modeling process that has been battle-tested and proven in the field. Luckily, you won't have to implement it from scratch!

Scikit-Learn comes with with a handy make_pipeline() function that helps you glue these steps together.

Page 7

2.8 - Pipeline dictionary

Finally, it's convenient to store the pipelines for each of your algorithms in a single dictionary.

This makes it easier to stay organized.

It also makes them easier to access by name later.

For example, we can start by adding the regularized regression algorithms into a dictionary, then the tree ensembles.