Generally speaking, overfitting occurs when out-of-sample (OOS) performance of some predictive system systematically differs from its performance in-sample (IS).
- From standard machine learning perspective, overfitting happens at the level of complexity where OOS performance starts to degrade as model complexity increases.
- In trading, they usually call overfitting any situation when OOS performance is worse than IS performance which is not sound theoretically.
It is clear why overfitting (both machine learning or trading definition) is dangerous in trading systems - it makes them unreliable and difficult to follow, not to speak about potential long-term losses. In papers section you may find research related to detection and prevention of overfitting when developing trading systems.