Strength in Numbers: Ensembling Models with Bagging and Boosting

Bagging and boosting are two powerful ensemble techniques in machine learning – they are must-knows for data scientists! After reading this article, you are going to have a solid understanding of how bagging and boosting work and when to use them. We’ll cover the following topics, relying heavily on examples to give hands-on illustration of the key concepts:

• How Ensembling helps create powerful models
• Bagging: Adding stability to ML models
• Boosting: Reducing bias in weak learners
• Bagging vs. Boosting – when to use each and why

Creating powerful models with ensembling

In Machine Learning, ensembling is a broad term that refers to any technique that creates predictions by combining the predictions from multiple models. If there is more than one model involved in making a prediction, the technique is using ensembling!

Ensembling approaches can often improve the performance of a single model. Ensembling can help reduce:

• Variance by averaging multiple models
• Bias by iteratively improving on errors
• Overfitting because using multiple models can increase robustness to spurious relationships

Bagging and boosting are both ensemble methods that can perform much better than their single-model counterparts. Let’s get into the details of these now!

Bagging: Adding stability to ML models

Bagging is a specific ensembling technique that is used to reduce the variance of a predictive model. Here, I’m talking about variance in the machine learning sense – i.e., how much a model varies with changes to the training dataset – not variance in the statistical sense which measures the spread of a distribution. Because bagging helps reduce an ML model’s variance, it will often improve models that are high variance (e.g., decision trees and KNN) but won’t do much good for models that are low variance (e.g., linear regression).

Now that we understand when bagging helps (high variance models), let’s get into the details of the inner workings to understand how it helps! The bagging algorithm is iterative in nature – it builds multiple models by repeating the following three steps:

1. Bootstrap a dataset from the original training data
2. Train a model on the bootstrapped dataset
3. Save the trained model

The collection of models created in this process is called an ensemble. When it is time to make a prediction, each model in the ensemble makes its own prediction – the final bagged prediction is the average (for regression) or majority vote (for classification) of all of the ensemble’s predictions.

Now that we understand how bagging works, let’s take a few minutes to build an intuition for why it works. We’ll borrow a familiar idea from traditional statistics: sampling to estimate a population mean.

In statistics, each sample drawn from a distribution is a random variable. Small sample sizes tend to have high variance and may provide poor estimates of the true mean. But as we collect more samples, the average of those samples becomes a much better approximation of the population mean.

Similarly, we can think of each of our individual decision trees as a random variable — after all, each tree is trained on a different random sample of the data! By averaging predictions from many trees, bagging reduces variance and produces an ensemble model that better captures the true relationships in the data.

Bagging Example

We will be using the load_diabetes¹ dataset from the scikit-learn Python package to illustrate a simple bagging example. The dataset has 10 input variables – Age, Sex, BMI, Blood Pressure and 6 blood serum levels (S1-S6). And a single output variable that is a measurement of disease progression. The code below pulls in our data and does some very simple cleaning. With our dataset established, let’s start modeling!

# pull in and format data
from sklearn.datasets import load_diabetes

diabetes = load_diabetes(as_frame=True)
df = pd.DataFrame(diabetes.data, columns=diabetes.feature_names)
df.loc[:, 'target'] = diabetes.target
df = df.dropna()

For our example, we will use basic decision trees as our base models for bagging. Let’s first verify that our decision trees are indeed high variance. We will do this by training three decision trees on different bootstrapped datasets and observing the variance of the predictions for a test dataset. The graph below shows the predictions of three different decision trees on the same test dataset. Each dotted vertical line is an individual observation from the test dataset. The three dots on each line are the predictions from the three different decision trees.

In the chart above, we see that individual trees can give very different predictions (spread of the three dots on each vertical line) when trained on bootstrapped datasets. This is the variance we have been talking about!

Now that we see that our trees aren’t very robust to training samples – let’s average the predictions to see how bagging can help! The chart below shows the average of the three trees. The diagonal line represents perfect predictions. As you can see, with bagging, our points are tighter and more centered around the diagonal.

We’ve already seen significant improvement in our model with the average of just three trees. Let’s beef up our bagging algorithm with more trees!

Here is the code to bag as many trees as we want:

def train_bagging_trees(df, target_col, pred_cols, n_trees):

    '''
        Creates a decision tree bagging model by training multiple 
        decision trees on bootstrapped data.

        inputs
            df (pandas DataFrame) : training data with both target and input columns
            target_col (str)      : name of target column
            pred_cols (list)      : list of predictor column names
            n_trees (int)         : number of trees to be trained in the ensemble

        output:
            train_trees (list)    : list of trained trees
    
    '''

    train_trees = []
    
    for i in range(n_trees):
        
        # bootstrap training data
        temp_boot = bootstrap(train_df)

        #train tree
        temp_tree = plain_vanilla_tree(temp_boot, target_col, pred_cols)

        # save trained tree in list
        train_trees.append(temp_tree)

    return train_trees

def bagging_trees_pred(df, train_trees, target_col, pred_cols):

    '''
        Takes a list of bagged trees and creates predictions by averaging 
        the predictions of each individual tree.
        
        inputs
            df (pandas DataFrame) : training data with both target and input columns
            train_trees (list)    : ensemble model - which is a list of trained decision trees
            target_col (str)      : name of target column
            pred_cols (list)      : list of predictor column names

        output:
            avg_preds (list)      : list of predictions from the ensembled trees       
        
    '''

    x = df[pred_cols]
    y = df[target_col]

    preds = []
    # make predictions on data with each decision tree
    for tree in train_trees:
        temp_pred = tree.predict(x)
        preds.append(temp_pred)

    # get average of the trees' predictions
    sum_preds = [sum(x) for x in zip(*preds)]
    avg_preds = [x / len(train_trees) for x in sum_preds]
    
    return avg_preds 

The functions above are very simple, the first trains the bagging ensemble model, the second takes the ensemble (simply a list of trained trees) and makes predictions given a dataset.

With our code established, let’s run multiple ensemble models and see how our out-of-bag predictions change as we increase the number of trees.

Admittedly, this chart looks a little crazy. Don’t get too bogged down with all of the individual data points, the lines dashed tell the main story! Here we have 1 basic decision tree model and 3 bagged decision tree models – with 3, 50 and 150 trees. The color-coded dotted lines mark the upper and lower ranges for each model’s residuals. There are two main takeaways here: (1) as we add more trees, the range of the residuals shrinks and (2) there is diminishing returns to adding more trees – when we go from 1 to 3 trees, we see the range shrink a lot, when we go from 50 to 150 trees, the range tightens just a little.

Now that we’ve successfully gone through a full bagging example, we are about ready to move onto boosting! Let’s do a quick overview of what we covered in this section:

1. Bagging reduces variance of ML models by averaging the predictions of multiple individual models
2. Bagging is most helpful with high-variance models
3. The more models we bag, the lower the variance of the ensemble – but there are diminishing returns to the variance reduction benefit

Okay, let’s move on to boosting!

Boosting: Reducing bias in weak learners

With bagging, we create multiple independent models – the independence of the models helps average out the noise of individual models. Boosting is also an ensembling technique; similar to bagging, we will be training multiple models…. But very different from bagging, the models we train will be dependent. Boosting is a modeling technique that trains an initial model and then sequentially trains additional models to improve the predictions of prior models. The primary target of boosting is to reduce bias – though it can also help reduce variance.

We’ve established that boosting iteratively improves predictions – let’s go deeper into how. Boosting algorithms can iteratively improve model predictions in two ways:

1. Directly predicting the residuals of the last model and adding them to the prior predictions – think of it as residual corrections
2. Adding more weight to the observations that the prior model predicted poorly

Because boosting’s main goal is to reduce bias, it works well with base models that typically have more bias (e.g., shallow decision trees). For our examples, we are going to use shallow decision trees as our base model – we will only cover the residual prediction approach in this article for brevity. Let’s jump into the boosting example!

Predicting prior residuals

The residuals prediction approach starts off with an initial model (some algorithms provide a constant, others use one iteration of the base model) and we calculate the residuals of that initial prediction. The second model in the ensemble predicts the residuals of the first model. With our residual predictions in-hand, we add the residual predictions to our initial prediction (this gives us residual corrected predictions) and recalculate the updated residuals…. we continue this process until we have created the number of base models we specified. This process is pretty simple, but is a little hard to explain with just words – the flowchart below shows a simple, 4-model boosting algorithm.

When boosting, we need to set three main parameters: (1) the number of trees, (2) the tree depth and (3) the learning rate. I’ll spend a little time discussing these inputs now.

Number of Trees

For boosting, the number of trees means the same thing as in bagging – i.e., the total number of trees that will be trained for the ensemble. But, unlike boosting, we should not err on the side of more trees! The chart below shows the test RMSE against the number of trees for the diabetes dataset.

This shows that the test RMSE drops quickly with the number of trees up until about 200 trees, then it starts to creep back up. It looks like a classic ‘overfitting’ chart – we reach a point where more trees becomes worse for the model. This is a key difference between bagging and boosting – with bagging, more trees eventually stop helping, with boosting more trees eventually start hurting!

With bagging, more trees eventually stops helping, with boosting more trees eventually starts hurting!

We now know that too many trees are bad, and too few trees are bad as well. We will use hyperparameter tuning to select the number of trees. Note – hyperparameter tuning is a huge subject and way outside of the scope of this article. I’ll demonstrate a simple grid search with a train and test dataset for our example a little later.

Tree Depth

This is the maximum depth for each tree in the ensemble. With bagging, trees are often allowed to go as deep they want because we are looking for low bias, high variance models. With boosting however, we use sequential models to address the bias in the base learners – so we aren’t as concerned about generating low-bias trees. How do we decide how the maximum depth? The same technique that we’ll use with the number of trees, hyperparameter tuning.

Learning Rate

The number of trees and the tree depth are familiar parameters from bagging (although in bagging we often didn’t put a limit on the tree depth) – but this ‘learning rate’ character is a new face! Let’s take a moment to get familiar. The learning rate is a number between 0 and 1 that is multiplied by the current model’s residual predictions before it is added to the overall predictions.

Here’s a simple example of the prediction calculations with a learning rate of 0.5. Once we understand the mechanics of how the learning rate works, we will discuss the why the learning rate is important.

So, why would we want to ‘discount’ our residual predictions, wouldn’t that make our predictions worse? Well, yes and no. For a single iteration, it will likely make our predictions worse – but, we are doing multiple iterations. For multiple iterations, the learning rate keeps the model from overreacting to a single tree’s predictions. It will probably make our current predictions worse, but don’t worry, we will go through this process multiple times! Ultimately, the learning rate helps mitigate overfitting in our boosting model by lowering the influence of any single tree in the ensemble. You can think of it as slowly turning the steering wheel to correct your driving rather than jerking it. In practice, the number of trees and the learning rate have an opposite relationship, i.e., as the learning rate goes down, the number of trees goes up. This is intuitive, because if we only allow a small amount of each tree’s residual prediction to be added to the overall prediction, we are going to need a lot more trees before our overall prediction will start looking good.

Ultimately, the learning rate helps mitigate overfitting in our boosting model by lowering the influence of any single tree in the ensemble.

Alright, now that we’ve covered the main inputs in boosting, let’s get into the Python coding! We need a couple of functions to create our boosting algorithm:

• Base decision tree function – a simple function to create and train a single decision tree. We will use the same function from the last section called ‘plain_vanilla_tree.’
• Boosting training function – this function sequentially trains and updates residuals for as many decision trees as the user specifies. In our code, this function is called ‘boost_resid_correction.’
• Boosting prediction function – this function takes a series of boosted models and makes final ensemble predictions. We call this function ‘boost_resid_correction_pred.’

Here are the functions written in Python:

# same base tree function as in prior section
def plain_vanilla_tree(df_train, 
                       target_col,
                       pred_cols,
                       max_depth = 3,
                       weights=[]):

    X_train = df_train[pred_cols]
    y_train = df_train[target_col]

    tree = DecisionTreeRegressor(max_depth = max_depth, random_state=42)
    if weights:
        tree.fit(X_train, y_train, sample_weights=weights)
    else:
        tree.fit(X_train, y_train)

    return tree

# residual predictions
def boost_resid_correction(df_train,
                           target_col,
                           pred_cols,
                           num_models,
                           learning_rate=1,
                           max_depth=3):
   '''
      Creates boosted decision tree ensemble model.
      Inputs:
        df_train (pd.DataFrame)        : contains training data
        target_col (str)               : name of target column
        pred_col (list)                : target column names
        num_models (int)               : number of models to use in boosting
        learning_rate (float, def = 1) : discount given to residual predictions
                                         takes values between (0, 1]
        max_depth (int, def = 3)       : max depth of each tree model

       Outputs:
         boosting_model (dict) : contains everything needed to use model
                                 to make predictions - includes list of all
                                 trees in the ensemble  
   '''



    # create initial predictions
    model1 = plain_vanilla_tree(df_train, target_col, pred_cols, max_depth = max_depth)
    initial_preds = model1.predict(df_train[pred_cols])
    df_train['resids'] = df_train[target_col] - initial_preds
    
    # create multiple models, each predicting the updated residuals
    models = []
    for i in range(num_models):
        temp_model = plain_vanilla_tree(df_train, 'resids', pred_cols)
        models.append(temp_model)
        temp_pred_resids = temp_model.predict(df_train[pred_cols])
        df_train['resids'] = df_train['resids'] - (learning_rate*temp_pred_resids)
        
    boosting_model = {'initial_model' : model1,
                      'models' : models,
                      'learning_rate' : learning_rate,
                      'pred_cols' : pred_cols}
    
    return boosting_model

# This function takes the residual boosted model and scores data
def boost_resid_correction_predict(df,
                                   boosting_models,
                                   chart = False):

   '''
      Creates predictions on a dataset given a boosted model.
      
      Inputs:
         df (pd.DataFrame)        : data to make predictions
         boosting_models (dict)   : dictionary containing all pertinent
                                    boosted model data
         chart (bool, def = False) : indicates if performance chart should
                                     be created
      Outputs:
         pred (np.array) : predictions from boosted model
         rmse (float)    : RMSE of predictions
   '''

    # get initial predictions
    initial_model = boosting_models['initial_model']
    pred_cols = boosting_models['pred_cols']
    pred = initial_model.predict(df[pred_cols])

    # calculate residual predictions from each model and add
    models = boosting_models['models']
    learning_rate = boosting_models['learning_rate']
    for model in models:
        temp_resid_preds = model.predict(df[pred_cols])
        pred += learning_rate*temp_resid_preds

    if chart:
        plt.scatter(df['target'], 
                    pred)
        plt.show()

    rmse = np.sqrt(mean_squared_error(df['target'], pred))

    return pred, rmse
    

Sweet, let’s make a model on the same diabetes dataset that we used in the bagging section. We’ll do a quick grid search (again, not doing anything fancy with the tuning here) to tune our three parameters and then we’ll train the final model using the boost_resid_correction function.

# tune parameters with grid search
n_trees = [5,10,30,50,100,125,150,200,250,300]
learning_rates = [0.001, 0.01, 0.1, 0.25, 0.50, 0.75, 0.95, 1]
max_depths = my_list = list(range(1, 16))

# Create a dictionary to hold test RMSE for each 'square' in grid
perf_dict = {}
for tree in n_trees:
    for learning_rate in learning_rates:
        for max_depth in max_depths:
            temp_boosted_model = boost_resid_correction(train_df, 
                                                        'target',
                                                         pred_cols, 
                                                         tree, 
                                                         learning_rate=learning_rate, 
                                                         max_depth=max_depth)
            temp_boosted_model['target_col'] = 'target'
            preds, rmse = boost_resid_correction_predict(test_df, temp_boosted_model)
            dict_key = '_'.join(str(x) for x in [tree, learning_rate, max_depth])
            perf_dict[dict_key] = rmse
            
min_key = min(perf_dict, key=perf_dict.get)
print(perf_dict[min_key])

And our winner is Read More