Grid search vs random search vs Bayesian optimization

Grid search enumerates all combinations over a Cartesian grid: wasteful when most hyperparameters don't matter. Random search samples uniformly: Bergstra & Bengio 2012 showed it matches/exceeds grid in high-dim spaces because not all hyperparameters matter equally. Bayesian optimization (Gaussian processes, TPE, Optuna) builds a surrogate model of validation-loss-as-a-function-of-hparams, picks the next trial by expected improvement: converges to good configs in far fewer trials. Hyperband, ASHA, BOHB combine BO with early stopping for cheap parallelism. Senior recipe 2025-2026: start with random search 20-50 trials to map the landscape, then switch to Optuna/TPE for fine-tuning; warm-start from past experiments. Always tune on a frozen eval set; never tune on test.