L1 vs L2 regularization

L2 (weight decay) adds a quadratic penalty on weights: lambda * sum(w^2). It shrinks weights smoothly toward zero, encouraging small-but-nonzero values. L1 adds an absolute-value penalty: lambda * sum(|w|). Because the L1 constraint region is a diamond (corners on axes), the optimum hits corners more often, shrinking some weights to exactly zero, producing sparsity, i.e., feature selection. Elastic Net combines both. When to pick: L2 if all features likely useful (deep nets, dense tabular); L1 if you want built-in feature selection (high-dim, low-signal feature spaces like text bag-of-words, genetics); ElasticNet when you want both shrinkage and selection. Senior nuance: modern deep net regularization (dropout, weight decay, label smoothing, MixUp) often outperforms strict L1/L2 in practice.