Gradient Boosting Tree (GBT)
First Principles Question
What if instead of averaging trees, each new tree specifically corrected the errors of all previous trees?
The Core Idea
GBT reframes boosting as gradient descent — but in function space instead of parameter space. At each step, you fit a new tree to the negative gradient of the loss function evaluated on the current predictions. This is equivalent to asking: “in which direction should my predictions move to reduce loss?”
The Math
Start with a constant prediction: F₀(x) = mean(y)
For m = 1 to M:
1. Compute pseudo-residuals (negative gradient):
rᵢₘ = -∂L(yᵢ, F(xᵢ))/∂F(xᵢ)
2. Fit a tree hₘ to residuals rᵢₘ
3. Update: Fₘ(x) = Fₘ₋₁(x) + α·hₘ(x)
α = learning rate (shrinkage)
Final model: F(x) = F₀ + α·h₁ + α·h₂ + ... + α·hₘ
GBT vs Random Forest
| Random Forest | GBT | |
|---|---|---|
| Trees built | In parallel | Sequentially |
| Each tree | Independent | Corrects previous |
| Fixes | Variance | Bias |
| Risk | Less | Overfits if too many trees |
| Speed | Faster | Slower |
Why Learning Rate Matters
Small α = more trees needed but better generalization. Large α = fewer trees but risks overfitting. This is identical to learning rate in gradient descent.
Prerequisites
Decision Tree · Optimization & Gradient Descent · Calculus
Builds To
XGBoost, LightGBM (engineered implementations of the same idea)
Content Ideas
Obsidian note: “Gradient Boosting is gradient descent. But instead of updating weights, you’re adding trees.”
X post: “GBT is just gradient descent where each step is a decision tree. That one reframe changes everything.”
GitHub: ml-from-scratch — GBT from scratch, visualize how residuals shrink each round