How to set max_depth and max_leaves in Python XGboost

Question

I have some questions regarding the max_depth and max_leaves hyperparams in XGBoost (I'm using Python).

Let's go through some examples.

Example 1. In the following case I have set:

max_depth = 3
max_leaves = 0

Note that I have also set some params for monotone_constraints.

model = XGBClassifier(
    objective='binary:logistic',
    base_score=0.5, 
    booster='gbtree', 
    colsample_bylevel=1,
    colsample_bynode=1, 
    colsample_bytree=1,
    enable_categorical=False, 
    gamma=2, 
    gpu_id=-1,
    importance_type=None, 
    interaction_constraints=[],
    learning_rate=0.09999999999999995, 
    max_delta_step=0,
    max_depth=3,
    max_leaves=0, #2^tree_depth 
    min_child_weight=0.9999999999999993, 
    monotone_constraints='(-1, -1, 1, 1)',
    n_estimators=12, 
    n_jobs=1, 
    nthread=1, 
    num_parallel_tree=1,
    predictor='auto',
    random_state=0, 
    reg_alpha=0.0009765625, 
    reg_lambda=1,
    scale_pos_weight=1, 
    silent=True, 
    subsample=1,
    tree_method='exact',
    validate_parameters=1, 
    pred_contribs=True,  
    verbosity=None)

The model is successfully trained and here is one of estimators (a tree):

Example 2.

In the following case I have set:

max_depth = 3
max_leaves = 0

Note that I have also set some params for monotone_constraints.

model = XGBClassifier(
    objective='binary:logistic',
    base_score=0.5, 
    booster='gbtree', 
    colsample_bylevel=1,
    colsample_bynode=1, 
    colsample_bytree=1,
    enable_categorical=False, 
    gamma=2, 
    gpu_id=-1,
    importance_type=None, 
    interaction_constraints=[],
    learning_rate=0.09999999999999995, 
    max_delta_step=0,
    max_depth=3,
    max_leaves=1, #2^tree_depth 
    min_child_weight=0.9999999999999993, 
    monotone_constraints='(-1, -1, 1, 1)',
    n_estimators=12, 
    n_jobs=1, 
    nthread=1, 
    num_parallel_tree=1,
    predictor='auto',
    random_state=0, 
    reg_alpha=0.0009765625, 
    reg_lambda=1,
    scale_pos_weight=1, 
    silent=True, 
    subsample=1,
    tree_method='exact',
    validate_parameters=1, 
    pred_contribs=True,  
    verbosity=None)

The model fails to train (the only error I get in Spyder is: Restarting Kernel ...). Question 1. Why does the model not get trained? Is there an optimal parameter for max_leaves given a certain parameter for max_depth? This error occurs for max_leaves >= 1 and max_leaves <= 7. The moment I set the max_leaves = 8 or higher the model is successfully trained. I don't understand why.

Example 3.

In the following case I have set:

max_depth = 0
max_leaves = 0

Note that I have also set some params for monotone_constraints.

model = XGBClassifier(
    objective='binary:logistic',
    base_score=0.5, 
    booster='gbtree', 
    colsample_bylevel=1,
    colsample_bynode=1, 
    colsample_bytree=1,
    enable_categorical=False, 
    gamma=2, 
    gpu_id=-1,
    importance_type=None, 
    interaction_constraints=[],
    learning_rate=0.09999999999999995, 
    max_delta_step=0,
    max_depth=0,
    max_leaves=0, #2^tree_depth 
    min_child_weight=0.9999999999999993, 
    monotone_constraints='(-1, -1, 1, 1)',
    n_estimators=12, 
    n_jobs=1, 
    nthread=1, 
    num_parallel_tree=1,
    predictor='auto',
    random_state=0, 
    reg_alpha=0.0009765625, 
    reg_lambda=1,
    scale_pos_weight=1, 
    silent=True, 
    subsample=1,
    tree_method='exact',
    validate_parameters=1, 
    pred_contribs=True,  
    verbosity=None)

In this case I get the error:

XGBoostError: [20:10:14] c:\windows\temp\abs_557yfx631l\croots\recipe\xgboost-split_1659548953302\work\src\tree\param.h:232: Max leaves and max depth cannot both be unconstrained.

Question 2. Why is that? Why do I have to set some parameters for max_depth and max_leaves?

Can anyone help me, please?

Answer 1

Leaves and depth are codependent.

XGBoost needs at least 2 leaves per depth, which means that it will need at least 2**n leaves, where n is depth. So for n=3, you would need at least 2**3=8 leaves. How can you imagine creating tree with depth 3 with just 1 leaf?

I suggest using specific package for hyperparameter optimization such as Optuna.