Prior probability and estimating your winrate

Suppose you move to Vegas to give it a go and try playing poker for a living. Live $1/2. Your first month you play for 200 hours and are up $20,000. Woo hoo! Pretty good, right? That's a winrate of $100/hr!

Now, would you go to your friends and say that you make $100/hr playing poker? Probably not, right? But why not? Well, The best live $1/2 grinders make maybe $30/hr. $100/hr at $1/2 is unheard of! So common sense says that you're just running hot. But let's try looking at this a little bit more deeply.

I'm going to talk about bayesian updating in this post, but I'm going to try to avoid technical language and actual numbers as much as possible. I want to convey that this sort of thinking is really just common sense.

Imagine that I asked you what you think your winrate is when you first moved to Vegas and haven't played a hand yet. Maybe you guess that it'd be $20/hr. Ok. That's your starting point. What the cool kids call, your prior.

From there, you play for a month and win $20,000 over 200 hours. This is data. Evidence. You take this, and you use it to update your belief.

It makes sense to update your beliefs based on the strength of the evidence you have. If you only played for a few days, that evidence isn't really strong, so you wouldn't want to update your beliefs too much. But here we have a whole month's worth of data. That's a lot, right? So with this strong evidence, maybe we want to update our beliefs a lot?

Well, that's one side of the coin. The other side of the coin is prior probability. Like I said in the beginning, a $100/hr winrate at $1/2 is unheard of. What does that mean? It means that you would need really really really strong evidence to justify believing that you actually are a $100/hr player. You may have heard "extraordinary claims require extraordinary evidence" before.

A month is solid, but it isn't nearly strong enough to outweigh the prior improbability of being a $100/hr player.

I want this to seem like common sense to you. At some level, I think it already is.

Imagine that you're watching a basketball game and the score says 992-88. Would you think to yourself, "Wow, the Blazers are really getting blown out tonight." No! You'd realize that it's a typo. And it would take a shit ton of evidence for you to actually believe that 992-88 is the real score.

Imagine that you see people talking about what a blow out the game is on Twitter. That's more evidence that 992-88 is the real score, but it only nudges you in that direction slightly. You're still highly skeptical.

Even if you saw a few articles online, you'd still probably only update a little bit. The journalists probably just made a mistake.

If you start seeing a major player like ESPN publish an article, you'd probably start scratching your head.

If you start seeing multiple reliable sources publish an article, maybe you start thinking that it actually was the real score.

Anyway, getting back to the point of this article, you can't just estimate your true winrate by looking at your actual winrate. Even though your actual winrate was $100/hr, you shouldn't estimate that to be your true winrate. You have to account for prior probability.

Here's how I would approach things. First, try to get acquainted with some sort of distribution of typical winrates. Something like this:

Then you have to try to figure out where you fall on that distribution. As you play more and more hands, that data starts to become pretty strong. But it takes a lot of hands to get to that point. Tens of thousands. Hopefully 100k+. What about before you get to that point?

I think it makes sense to start incorporating more "subjective" data. For example, how much do you study? And how much do your opponents study? If you're studying a lot and your opponents aren't, then I think that's evidence that you can use to shift your estimate.

Same with if you have more experience than them. Or if you review all of your hands and find that you're making the right plays, and you're discussing them with good players who also think you're making the right plays. I think that all of that sort of stuff is evidence that you can use to shift your estimate of where your true winrate is.

Notice that I put "subjective" in quotes earlier. Some people might say that this is all wishy-washy and you can't use this sort of stuff as evidence. All I can say to that is that it's wrong. Anything that nudges your belief one way or the other is evidence. Maybe it only nudges you a little bit. In that case it'd be weak evidence, but it still would be evidence. See Scientific Evidence, Legal Evidence, Rational Evidence.

Some parting advice: beware of overconfidence and bias. Don't take this as an excuse to say, "I study poker, my opponents stuck, I know I've been breaking even for six months but I still think I'm a way better player than all of them."