Another Way of Growing Trees: XGBoost's grow_policy

In addition to limiting the maximum depth of trees using a max_depth hyperparameter, there is another paradigm for controlling tree growth: finding the node where a split would result in the greatest reduction in the loss function, and splitting this node, regardless of how deep it will make the tree. This may result in a tree with one or two very deep branches, while the other branches may not have grown very far. XGBoost offers a hyperparameter called grow_policy, and setting this to lossguide results in this kind of tree growth, while the depthwise option is the default and grows trees to an indicated max_depth, as we've done in Chapter 5, Decision Trees and Random Forests, and so far in this chapter. The lossguide grow policy is a newer option in XGBoost and mimics the behavior of LightGBM, another popular gradient boosting package.

To use the lossguide policy, it is necessary to set another hyperparameter we haven...