All of value betting rests on one assumption: your probabilities are accurate. A model is well calibrated if, among all the events it assigns 60% to, about 60% actually happen.
Accuracy ≠ calibration
- A model can rank teams correctly (good accuracy) while giving probabilities that are too extreme (85% where reality is 70%).
- For Kelly, that's catastrophic: overly extreme probabilities generate false EVs and oversized stakes.
The Brier score
where equals 1 if the event happened, 0 otherwise. Lower is better. A constant 50% forecast gives 0.25.
Log loss
It punishes very harshly confident but wrong forecasts (calling 95% for an event that doesn't happen). It's the metric most consistent with Kelly, since Kelly also maximises a logarithmic expectation.
The calibration curve
- Group your forecasts into bins (0-10%, 10-20%, …).
- For each bin, calculate the actual frequency of occurrence.
- Plot actual frequency (vertical axis) against forecast probability (horizontal axis).
A perfect model follows the diagonal. A curve that's "flatter" than the diagonal means your model is overconfident.
Interactive tool
Perfect calibration
Educational tool. Results are theoretical estimates, not betting advice.
The decisive test: beating the market
The real judge isn't absolute calibration, but the comparison with the devigged market:
- Calculate your model's log loss and that of the market probabilities on the same events.
- If your model does no better than the market, its "value bets" are probably noise.
- If you do slightly better on certain segments (a league, a type of market), that's where your value lies.
Recalibrating
If your model is poorly calibrated, simple methods exist:
- Platt scaling (logistic regression on the model's outputs).
- Isotonic regression.
- Blending with the market (the shrinkage from the robust Kelly module), estimating the weight by minimising log loss.