Ensembles and Random Forests

Goal

By the end of this lesson, you can explain why combining diverse trees can be more stable than relying on one tree, and describe bootstrap sampling, random feature subsets, and voting in a Random Forest.

Ask several imperfect models instead of one

If several people make different estimation errors, their average can be more stable than one person's guess. If they all make exactly the same error, averaging does not help much.

An ensemble applies the same idea to models: combine several predictors so one fitted model's mistake does not automatically decide the final answer.

Bagging creates different training views

A Random Forest trains many decision trees. For each tree, it draws a bootstrap sample: training rows sampled from the original training set with replacement.

Some original rows appear more than once and some do not appear in that tree's sample.

This is a form of bagging, short for bootstrap aggregating.

Random feature subsets add more diversity

At each split, a Random Forest tree considers only a random subset of features rather than always considering every feature.

That makes it harder for one strong feature to force all trees into nearly identical structures.

The goal is useful diversity among reasonably capable trees.

Combine predictions

For classification, trees vote or their class probabilities are averaged. For regression, numeric predictions are averaged.

A noisy mistake from one tree may be outvoted by the others.

This does not make forests immune to leakage, biased data, or distribution shift. An ensemble can learn the wrong problem very consistently.

Compare one tree with a forest

1. Run the Lab on the same fixed train/held-out split.
2. Compare the single tree and Random Forest scores.
3. Inspect bootstrap, random feature selection, and the fixed seed.
4. Change the number of trees from 25 to 5.
5. Predict whether the smaller forest will be more sensitive to which trees happened to be sampled.
6. Run again, then restore 25 trees and change the random seed.

Loading lab…

The lesson is not “Random Forest always beats one tree.” The lesson is that averaging diverse trees can reduce the instability of one fitted tree.

Random Forest versus one deep tree

A single deep tree can produce an irregular boundary and may fit training noise.

A Random Forest can still use expressive trees, but the final prediction aggregates many differently trained trees. That often reduces variance while keeping nonlinear modeling power.

The tradeoff is that the forest is harder to explain as one simple sequence of rules.

Quick Check

1. What is a bootstrap sample?

A copy of the held-out test setA list containing every feature exactly onceA sample of training rows drawn with replacement

2. Why does Random Forest consider random feature subsets?

To guarantee every tree is shallowTo leak less test informationTo encourage trees to make different useful errors

3. How are Random Forest classification predictions combined?

By choosing whichever single tree has the highest training scoreBy sequentially correcting the previous treeBy aggregating votes or class probabilities from many trees

0 of 3 questions answered.

Key Takeaways

• Ensembles combine multiple predictors rather than trusting one fitted model.
• Bagging trains models on bootstrap samples of the training rows.
• Random Forest also uses random feature subsets to increase tree diversity.
• Classification forests aggregate votes or probabilities; regression forests average values.
• Diversity matters because identical trees would repeat the same errors.

Next Lesson

Next, you will contrast Random Forest with boosting, where later trees are trained to reduce errors left by earlier trees.

References

• scikit-learn, Ensemble methods.
• scikit-learn, RandomForestClassifier.