Model Validation
Model Performance Metrics and Diagnostics on Out-of-Sample Data
15.3%
Better than Player Historical Avg
2.5x
Top vs Bottom Decile Lift
42824
Player Appearances
0.153
Prediction Correlation
Calibration: Predicted vs Actual Scoring Rates
When we predict a 30% chance of scoring, does it happen ~30% of the time? Perfect calibration follows the diagonal line.
Ranking Performance: Scoring Rate by Prediction Decile
Players ranked in the top decile by our predictions score 27.5% of the time, compared to 11.0% for the bottom decile - a 2.5x lift.
Brier Score Comparison
Lower is better. Brier Score measures the accuracy of probabilistic predictions.
| Model | Brier Score | vs SFM |
|---|---|---|
| SFM | 0.0598 | - |
| Naive | 0.0706 | +15.3% |
Naive: Poisson distribution using each player's historical scoring rate from training data.
Based on 27599 player appearances with training history.
Performance by League
SFM consistently outperforms the Naive model across all 5 major European leagues.
| League | Impr. | SFM |
|---|---|---|
| Premier League | +11.5% | 0.0608 |
| La Liga | +19.7% | 0.0572 |
| Serie A | +16.0% | 0.0561 |
| Bundesliga | +10.9% | 0.0632 |
| Ligue 1 | +16.8% | 0.0635 |
| Champions League | +22.9% | 0.0575 |
Detailed Decile Analysis
Players sorted by predicted scoring probability, split into 10 equal groups.
| Decile | Predicted | Actual |
|---|---|---|
| 1 (Bottom) | 8.0% | 11.0% |
| 2 | 10.7% | 12.1% |
| 3 | 12.2% | 11.0% |
| 4 | 13.1% | 8.9% |
| 5 | 13.9% | 12.1% |
| 6 | 15.2% | 12.1% |
| 7 | 16.5% | 12.5% |
| 8 | 17.5% | 12.9% |
| 9 | 19.8% | 18.6% |
| 10 (Top) | 28.6% | 27.5% |
Performance by Season
SFM maintains strong performance across different seasons.
| Season | Impr. | SFM |
|---|---|---|
| 2025/26 | +24.3% | 0.0513 |
| 2024/25 | +2.4% | 0.0706 |