Model Validation
Model Performance Metrics and Diagnostics on Out-of-Sample Data
The model trains through the last complete season, so nothing is held back mid-sample.
Its out-of-sample performance is measured by scoring a sealed season exactly
once, under a pre-registered protocol — no re-runs, no re-opening. That result
is below, reported as-is.
+3.56%
Skill vs. naive
paired t = 12.0
0.50742
Log-loss
naive 0.52616
60.21%
geoP
naive 59.09%
18996
Appearances scored
2024/25 · 1463 new players
Reading the numbers.
- Skill vs. naive — how much the model improves on a naive forecast that just uses each outcome's overall frequency. Positive means it beats it.
- Log-loss — the standard score for probabilistic forecasts. It punishes confident wrong calls hardest. Lower is better.
- geoP — the typical probability the model put on what actually happened. A player either scores 0, 1, 2 or 3+ in a match; geoP 60.21% means that on an average appearance the model gave about 60.21% to the bucket that occurred. It is exactly exp(−log-loss), so it says the same thing as log-loss on a scale you can read as a probability. Higher is better.
- Appearances scored — how many player-matches the holdout covers. Bigger means the result is less likely to be luck.
What this number is — and isn't
- The 2024/25 season was sealed and scored once, on 2026-08-14, by SFM II. Per protocol the result is recorded, not acted on.
- Cross-validation had estimated +4.56% (folds 4.3–4.96%). The holdout came in about a point lower — the usual selection shrinkage from comparing many variants on those folds, plus single-season variation. The holdout is the honest estimate, not the cross-validation figure.
- Weakest segment: mid-season (matchdays 11–20).
- 2025/26 remains sealed as the pristine holdout for the next cycle.
| Segment | SFM log-loss | Naive | Note |
|---|---|---|---|
| Cold start (matchdays 1–2) | 0.4891 | 0.5068 | strongest — the model earns its keep before form data exists |
| New players | 0.5362 | 0.505 | harder: 1,463 rows with no history to lean on |
This is the model's validated skill before the season starts. The SFM Tracker records how the published picks actually fare, match by match, as results come in.