Overfitting occurs when a model learns the training data too well, including its noise and specific nuances, leading to poor performance on new, unseen data. This is characterized by high variance. Underfitting, conversely, happens when a model is too simple to capture the underlying patterns in the data, resulting in poor performance on both training and new data. This is characterized by high bias.
Complex models with many parameters, such as deep decision trees, random forests, and neural networks, are more prone to overfitting because they have the capacity to memorize the training data. Simple models like linear regression or logistic regression, especially with limited features, are more susceptible to underfitting as they may not have enough flexibility to model complex relationships.
For instance, a decision tree can overfit if it's allowed to grow to its maximum depth, creating very specific rules for individual data points. Linear models might underfit if the true relationship in the data is non-linear and the model is constrained to a linear form.
Nvidia
OpenAI
Reddit