SV

Street Verdict

Analyst Scorecard

All of Learn

Testing an idea 2: reading results

Drawdown, win rate, Sharpe, walk-forward: what each number means and when a result is too good to be true.

0 of 9 done

All modulesModule 7 of 9

Overfitting

Try 50 settings on the early years, keep the top scorer, then see how it did on the later years it never saw. Repeat on several stocks.

Do this first

Try 50 settings, keep the top scorer

The EMA crossover has two knobs: the length of the fast average and the slow one. The experiment tries 50 combinations on the early part of the stock's history (the first 70% of up to ten years), keeps whichever made the most after costs, and only then looks at how all 50 did on the later 30% that played no part in the choice.

5/85/105/155/205/2510/1510/2010/3010/4010/5015/2315/3015/4515/6015/7520/3020/4020/6020/8020/10025/3825/5025/7525/10025/12530/4530/6030/9030/12030/15040/6040/8040/12040/16040/20050/7550/10050/15050/20050/25060/9060/12060/18060/24060/30080/12080/16080/24080/32080/400

Up to the last 10 years of daily prices, ₹1,00,000 per trade, real charges and slippage.

Read it

Why the top scorer of many disappoints

Overfitting is fitting a strategy to the noise of one stretch of history rather than to anything that repeats. Every setting's result is part skill (if there is any) and part luck: which trends happened to come along, which crossovers happened to land on a good day. When you try 50 settings and keep the top scorer, you are mostly keeping the luckiest. The luck does not come with it into the future.

The part of history used to choose is called in-sample; the part held back is out-of-sample. Only the out-of-sample result estimates what choosing this way would really have earned. The gap between the two is the price of the search.

How to protect yourself.Use few knobs and round numbers. Decide what you will test before you look. Set aside the most recent years and look at them once, at the end. Prefer settings whose neighbours also did well (a lone spike on the grid is luck). And use a walk-forward test, next module, which repeats the hold-back month after month.

Repeat it

Does it happen on other stocks too?

One stock could be a fluke. Run the identical experiment on eight large NSE stocks (yours first) and count how often the chosen setting held up.

Check yourself

0 of 3 right

  1. 1. You try 50 settings on the same history and keep the top scorer. Why is its result probably too good?

  2. 2. What is out-of-sample data?

  3. 3. The top-scoring in-sample setting did much worse on the later years. What is the honest estimate of the strategy?