introduction

Página 1

XGBoost: A Scalable Tree Boosting System

supports both regression and classification predictive modeling problems

(eXtreme Gradient Boosting)

Gradient boosting :

Idea : fitting a new model to the gradient of the error of the updated sum of models

While stochastic gradient descent keeps the model as it is and tunes the param to minimise the loss,

the architecture of the model is fixed. What you are therefore trying to optimize are the parameters, P of the model (in logistic regression, this would be the weights)

I am trying to find the best parameters P for my function F, where ‘best’ means that they lead to the smallest loss possible

F(x|p) = min loss (p)(y, f(x|p)

While gradient boosting doesn’t assume the fixed architecture

F(x|p) = min loss(p,F)(y, f(x|p)The only thing that has changed is that now, in addition to finding the best parameters P, I also want to find the best function F.

GB taking lots of simple functions, and adding them together

Where SGD trains a single complex model, gradient boosting trains an ensemble of simple models.

The next model I am going to fit will be on the gradient of the error with respect to the predictions, ∂Loss/∂ŷ .

2. Gradient Boosted Trees

Gradient boosted trees consider the special case where the simple model h is a decision tree.

AS compared to GBM , xgboost used a more regularized model formalization to control over-fitting, which gives it better performance

implementation of gradient boosted decision trees designed for speed and performance.

the library is laser focused on computational speed and model performance,

Three main forms of gradient boosting are supported:

Gradient Boosting algorithm also called gradient boosting machine including the learning rate.

Stochastic Gradient Boosting with sub-sampling at the row, column and column per split levels.

Regularized Gradient Boosting with both L1 and L2 regularization.

Some key algorithm implementation features include:

Sparse Aware implementation with automatic handling of missing data values.

Block Structure to support the parallelization of tree construction.

Continued Training so that you can further boost an already fitted model on new data

XGBoost is fast. Really fast when compared to other implementations of gradient boosting.

Performance wise: XGBoost dominates structured or tabular datasets on classification and regression predictive modeling problems.

ALGORITHM:

XGBoost library implements the gradient boosting decision tree algorithm.

Boosted Trees(GBM) tries to add new trees that compliments the already built ones. This normally gives you mbetter accuracy with less trees than random forests.

(Boosting is an ensemble technique where new models are added to correct the errors made by existing models. Models are added sequentially until no further improvements can be made. A popular example is the AdaBoost algorithm that weights data points that are hard to predict.)

Gradient boosting is an approach where new models are created that predict the residuals or errors of prior models and then added together to make the final prediction. It is called gradient boosting because it uses a gradient descent algorithm to minimize the loss when adding new models.

Memory optimization, most memory construction are done in first pass, and no dynamic memory is involved.

- Cache-line optimization, the training pattern tries to be cache-friendly in some sense.

- The improvement in terms of model itself, we develop variant of the model to make the algorithm more robust, and more accurate in general, see Introduction to Boosted Trees for introduction.

Página 2

DEEP LEARNING VS XGBOOST

A deep net can capture things like image, audio and possibly text quite well by modeling the spatial temporal locality. While tree based models(the one that XGBoost is good at) solves tabular data very well and have some certain properties a deep net does not have (e.g. being easier to interpret and invariant to input scale, much easier to tune).