The Bias-Variance Trade-Off
Understanding Overfitting
Simulation Controls
Complexity of Truth (f)
Linear
Non-Linear
Highly Curvy
Model Flexibility (df)
Linear (1 df)
5 df
High (25 df)
Noise Level (Var(ϵ))
Low
4.0
High
Regenerate Training Data
Bias-Variance Rules
Training Error (Gray):
Always decreases as flexibility increases. The model "memorizes" the specific training points.
Test Error (Red):
Initially decreases as bias drops, but eventually increases as variance dominates.
Irreducible Error:
The golden dashed line. No model can ever perform better than this.
Fitted Model (f̂*) vs. Truth (f)
MSE Curve Comparison