Model calibration

All of value betting rests on one assumption: your probabilities are accurate. A model is well calibrated if, among all the events it assigns 60% to, about 60% actually happen.

Accuracy ≠ calibration

• A model can rank teams correctly (good accuracy) while giving probabilities that are too extreme (85% where reality is 70%).
• For Kelly, that's catastrophic: overly extreme probabilities generate false EVs and oversized stakes.

The Brier score

where $y_i$ equals 1 if the event happened, 0 otherwise. Lower is better. A constant 50% forecast gives 0.25.

Log loss

It punishes very harshly confident but wrong forecasts (calling 95% for an event that doesn't happen). It's the metric most consistent with Kelly, since Kelly also maximises a logarithmic expectation.

The calibration curve

1. Group your forecasts into bins (0-10%, 10-20%, …).
2. For each bin, calculate the actual frequency of occurrence.
3. Plot actual frequency (vertical axis) against forecast probability (horizontal axis).

A perfect model follows the diagonal. A curve that's "flatter" than the diagonal means your model is overconfident.

The decisive test: beating the market

The real judge isn't absolute calibration, but the comparison with the devigged market:

• Calculate your model's log loss and that of the market probabilities on the same events.
• If your model does no better than the market, its "value bets" are probably noise.
• If you do slightly better on certain segments (a league, a type of market), that's where your value lies.

Recalibrating

If your model is poorly calibrated, simple methods exist:

• Platt scaling (logistic regression on the model's outputs).
• Isotonic regression.
• Blending with the market (the shrinkage from the robust Kelly module), estimating the weight $w$ by minimising log loss.