Machine learning algorithms perform much more than just forecasting. The accuracy of machine learning results cannot be improved simply by fitting data into models. As the data gets large and complex, better data handling techniques are eminent to handle it.
Also read: What is Clustering in Machine Learning: Types and Methods
Gradient Boosting is an ensemble technique. It is primarily used in classification and regression tasks. It provides a forecast model consisting of a collection of weaker prediction models, mostly decision trees.
In this article, we will deep dive into the details of gradient-boosting algorithms in machine learning. Besides looking at what is gradient boosting we will also learn about gradient boosting models, and types of gradient boosting, and look at its examples as well.
What is Gradient Boosting?
Gradient Boosting is a type of machine learning boosting technique. It builds a better model by merging earlier models until the best model reduces the total prediction error. Also referred to as a statistical forecasting model, the main idea of gradient boosting is to attain a model that eliminates the errors of the previous models.
Related to this: What is Cost Function in Machine Learning
In each case, the target outcomes in the data decide the effect a case’s forecast has on the overall prediction inaccuracy. There are two possible cases for this:
Gradient Boosting is named so that the set target outcomes depend on the gradient of the inaccuracy vs the forecast. Every new model created using this method moves closer to the path that lowers prediction error in the range of potential outcomes for every ML training case.
Gradient Boosting is mainly of two types depending on the target columns:
- Gradient Boosting Regressor: It is used when the columns are continuous
- Gradient Boosting Classifier: It is used when the target columns are classification problems
The “Loss Function” acts as a distinguisher for them. It is among the three main elements on which gradient boosting works.
- Loss Function: The primary goal in this situation is to maximize the loss function, which is not constant and changes according to the problems. It is simple to create one’s own standard loss function, however, it must be differentiable.
- Weak Learners: These are used mainly for predictions. A decision tree is an example of weak learners. For the real output values needed for splits, specific regression trees are applied.
- Additive Model: There are more trees added at once, but no changes are made to the model’s already-existing trees. A gradient descent approach reduces the losses when the trees are added.
Gradient Boosting Algorithm Examples
Let us understand how Gradient Boosting algorithms work through an example. In this case, we will have a continuous target column, thus using Gradient Boosting Regressor .
We will use a random dataset with different features. We have to predict target values, while all other characteristics are standalone features. We need to observe if the learning algorithm is able to figure out the irrelevant characteristics.
Step 1
The first step includes building a base model to predict the dataset. For easy calculations, we will take an average of the target column and Assume that is the anticipated value, as displayed below:
Mathematical interpretation of the 1st step:
Where, L= Loss Function
Gamma= Predicted Value
Argmin= predicted value or gamma to be found of which the loss function is minimum
Furthermore, as our target column is continuous , the Loss function is to be calculated by:
Step 2
Find the pseudo residuals (observed value – predicted value)
The predicted value in this is the value forecasted in the prior model. Because our target column is now incorrect, it should be noted that the forecasts in this situation will reflect the error values rather than the expected car price values.
Step 3
This step includes finding the output values of each leaf of the decision tree . We have to find the output for all the leaves since a single leaf might get more than one residue. No matter if there is just one number or more, we can easily calculate the result by taking the average of all the values in a leaf.
Mathematical expression:
Where, hm(xi)= DT produced during residual
m= no. of DT
Step 4
The final step involves the update of the forecasts of the previous model.
Mathematical Expression for updating the model:
Where,
M= no. of decision trees made
Fm-1(x)= forecasts of the base model
The model has been explained through a gradient-boosting example in the below video: