Random Forest
First Principles Question
If one decision tree overfits badly, what happens when you average 100 of them?
The Core Idea
A single decision tree has high variance — it changes dramatically with small changes in training data. But if you train many trees on different random subsets of data and features, and average their predictions, the variance cancels out. The key insight: errors must be uncorrelated for averaging to help. Random Forest forces this with two randomization tricks.
The Two Tricks
- Bootstrap sampling: each tree trains on a random sample with replacement from the training data (~63% unique samples per tree)
- Feature subsampling: at each split, only a random subset of features is considered (typically √n_features)
These two tricks decorrelate the trees so their errors don’t all point the same direction.
Why It Works (Bias-Variance)
- Individual trees: low bias, high variance
- Averaging uncorrelated models: variance drops, bias stays the same
- Result: low bias, lower variance → better generalization
Bagging vs Random Forest
Bagging uses all features at each split. Random Forest adds feature subsampling — this is the key difference that decorrelates the trees further.
Out-of-Bag Error
~37% of samples not used in each tree’s bootstrap sample. These “out-of-bag” samples can be used as a built-in validation set — no cross-validation needed.
Prerequisites
Decision Tree · Statistics & Bias-Variance
Builds To
Gradient Boosting Tree — a different approach to fixing the same tree problem
Content Ideas
Obsidian note: “Why averaging bad models produces a good one — the math behind Random Forest.”
X post: “Random Forest takes 100 overfit models and averages them into a good one. Here’s why that works.”
GitHub: ml-from-scratch — implement bagging from scratch, show variance reduction curve