SV

Street Verdict

Analyst Scorecard

Judging market advice

Lesson 7.3 · about 10 minutes

Luck versus skill

  • Hypothetical

Learn to

  • See a great record appear from pure chance
  • Ask how many people tried before asking how good the top one is
  • Expect a run of luck to fade on the next set of calls

Ideas in this lesson

  • Selection
  • Survivorship
  • Many tries
  • Regression to the middle

Everything on this page is hypothetical. No real analyst appears in it. One hundred imaginary analysts each make the same number of calls, and every call is decided by a fair coin: half hit, half miss, with no skill anywhere.

With enough people trying, some will still string together a run that looks impressive. If you only meet the ones at the top, you will think skill is at work. This is the heart of why a good-looking record needs two questions: how many people were trying, and what did the record do next?

Run it, look at the five top records, then let those same five make a fresh set of calls. Change the seed to deal a new set of one hundred.

Try it · hypothetical data

One hundred hypothetical analysts, a coin for every call. Look at the top five, then watch what happens next.

Hypothetical: a fair coin decides every call. No real analyst is shown.

Seed 1. The same seed always gives the same hundred.

First run: the top five of one hundred (100 hypothetical analysts, 10 calls each)

0510

Hits out of 10 on the bottom, number of analysts as the height of each bar.

  • Analyst 2

    9 of 10

    90% hit rate

  • Analyst 56

    8 of 10

    80% hit rate

  • Analyst 69

    8 of 10

    80% hit rate

  • Analyst 81

    8 of 10

    80% hit rate

  • Analyst 8

    7 of 10

    70% hit rate

A coin gets 9 or more right out of 10 about 1.1% of the time. Among 100 analysts that is about 1.1 of them by luck alone. How many analysts were there? That is the question a single great record never answers.

Loading real graded calls…

The stars did not get worse. They were never better. Their first run was luck and the second run is what a coin does. Real markets are not coins, and some people are skilled. The point is narrower: a record seen without the number of tries behind it cannot tell luck from skill.

That is also why a stack of backtests or a long list of past picks is weak evidence on its own. The Testing an idea track shows the same effect on trading rules, under the name overfitting.

Check yourself

0 of 3 right

  1. 1. Of 1,000 fair-coin analysts, each making 10 calls, about how many show 9 or more right?
  2. 2. You are shown the top record among analysts you have never heard of. What is the first question to ask?
  3. 3. The top five coin analysts make 10 more calls each. What should their average be?

Answer every question correctly to complete this lesson.

Lessons use prices at least 30 days old. The live pages show current ones.

Sources

Last reviewed 30 Oct 2026 by the author. Independent fact review is pending before public launch. Facts are as of the dates shown. Education only; not investment advice.