Bagging vs boosting vs Random Forest

Bagging (e.g., Random Forest) trains M models on bootstrap samples and averages - it reduces variance without changing bias much, so leaves the model class unchanged. Boosting (AdaBoost, GBM, XGBoost, LightGBM) fits models sequentially, each one up-weighting previous errors, trading variance for bias and producing a strong learner from weak base learners. Random Forest = bagging + at each split a random subset of features - the feature bagging decorrelates trees and dramatically lowers variance. Use RF when you need a robust, low-tuning baseline; use gradient boosting when leaderboard performance matters but watch for overfitting on small data and the resulting need for early stopping, shrinkage, and column subsampling.