When should you trust a probabilistic forecast?

01 — Define the claim

A probability is not a disguised yes-or-no answer.

A 60% forecast does not promise that this particular event will happen. It makes a statement about repeated behaviour: across a sufficiently large set of comparable 60% forecasts, the event should occur about 60% of the time. That distinction changes how the system has to be evaluated.

Presaira therefore treats the output as a distribution rather than a single predicted scoreline. Brier score and log loss sit ahead of top-pick accuracy because they score the probabilities themselves. A confidently wrong distribution should be punished more than a cautious miss, and a well-calibrated probability should get credit even when one individual outcome goes the other way.

This is also why a single spectacular prediction is weak evidence. Probabilistic quality is a property of a collection of forecasts, not a screenshot of one correct call.

02 — Score honestly

Use proper scores, then inspect reliability instead of worshipping one number.

A proper scoring rule gives the forecaster an incentive to state the probability it actually believes. In the Presaira evaluation, multiclass Brier score and log loss provide that baseline discipline. A uniform one-third model is also kept visible as a deliberately simple reference rather than letting the model grade itself in isolation.

But a Brier score alone is not a complete trust certificate. Reliability matters: probabilities near 0.6 should verify near 60% over enough observations. Sharpness matters too: a model that always predicts the base rate can look reliably cautious while contributing little decision value. Reliability diagrams help separate those behaviours, provided the bins contain enough observations to mean something.

Tournament samples are small. Presaira explicitly notes that calibration bins at the tails can be dominated by one or two matches. A diagram with sparse bins should reduce confidence in the conclusion, not invite stronger storytelling.

• Score the full probability vector, not only the winning class.
• Keep a simple reference model visible.
• Read calibration together with bin counts and sample size.

03 — Protect the timeline

A clean metric is worthless if future information leaked into the past.

Temporal leakage is one of the easiest ways to make a forecasting model look better than it was. The training boundary has to be an executable rule, not a sentence in a README. Presaira records a 2026-06-10 training cutoff for the published tournament run and contains a pre-cutoff assertion that raises when future rows enter the training set.

The historical evaluation separates tuning from holding out: 2022 was used for tuning while 2018 remained held out for evaluation. That is more informative than reporting a score from a dataset that influenced every modeling choice.

The general test is simple: if I cannot explain what information was available at the moment each forecast would have been issued, I do not yet have an honest out-of-sample result.

04 — Count every outcome

Coverage is part of the metric.

A good average can be manufactured by quietly dropping awkward observations. For the 2026 tournament, Presaira records 72 of 72 group matches and 32 of 32 knockout matches, with an empty omitted list. That coverage statement belongs beside the score because the score is only meaningful over the set it actually represents.

The same principle applies well beyond sport. If a pipeline rejects difficult records, if a sensor disappears during bad conditions, or if an evaluation excludes cases after seeing their outcomes, the denominator becomes a modeling decision. Report it explicitly.

05 — Calibrate only when evidence earns it

Calibration is a model choice, not a ceremonial final step.

Post-hoc calibration can improve probability quality, but it can also overfit a small calibration window. In Presaira, Platt scaling and isotonic regression were both tested. Platt was effectively neutral on Brier while degrading log loss; isotonic worsened the 2022 Brier score by 2.3%. The recorded decision was therefore calibration_method: none.

That result matters because 'no calibration' can mean two very different things: nobody checked, or calibration methods were evaluated and the evidence did not justify adding one. Only the second is a defensible engineering decision.

A trustworthy pipeline should make rejected interventions visible. Otherwise the published model becomes a survivor selected from an invisible garden of experiments.

06 — Reproduce the system

Reproducibility includes simulation, versions and failed checks.

Tournament probabilities depend on more than the match model. Group rules, knockout progression, third-place qualification, random sampling and simulation count all affect the final distribution. Presaira runs the tournament 50,000 times with a fixed seed and records model and simulator versions so the published artifact has an inspectable computational context.

One of the most useful pieces of evidence is a failed convergence check: the committed 10,000-versus-50,000 target does not pass its chosen threshold, so production stays at 50,000. Keeping that failure is stronger evidence than retroactively choosing a threshold that makes the run look clean.

07 — Bound the trust

A model can be useful without pretending to be an oracle.

Presaira does not see team news, injuries, suspensions or lineups. It was also under-confident in parts of the 2026 probability range, and the historical backtests sit within one standard deviation of a strong Elo-based baseline. Those limitations are not footnotes to remove from the story; they define where the forecast deserves reliance.

My practical trust test is therefore cumulative. I want proper scoring, calibration evidence, complete coverage, leakage protection, reproducible artifacts and explicit limitations. None proves a forecast is 'correct'. Together they make it possible to understand what the numbers actually deserve.