Model Validation

Model Performance Metrics and Diagnostics on Out-of-Sample Data

17.3%

Better than Player Historical Avg

2.56x

Top vs Bottom Decile Lift

54555

Player Appearances

0.152

Prediction Correlation

Methodology: All metrics use multi-class Brier score comparing full goal distributions (0, 1, 2, 3+ goals) on held-out test data (54555 player appearances). The Naive model uses each player's Poisson distribution derived from their historical scoring rate in the training data. Both SFM and Naive are evaluated on the same task: predicting the complete goal probability distribution, not just "score vs not score."
Calibration: Predicted vs Actual Scoring Rates

When we predict a 30% chance of scoring, does it happen ~30% of the time? Perfect calibration follows the diagonal line.

Ranking Performance: Scoring Rate by Prediction Decile

Players ranked in the top decile by our predictions score 26.9% of the time, compared to 10.5% for the bottom decile - a 2.56x lift.

Brier Score Comparison

Lower is better. Brier Score measures the accuracy of probabilistic predictions.

Model Brier Score vs SFM
SFM 0.058 -
Naive 0.0702 +17.3%

Naive: Poisson distribution using each player's historical scoring rate from training data.

Based on 32724 player appearances with training history.

Performance by League

SFM consistently outperforms the Naive model across all 5 major European leagues.

League Impr. SFM Naive Apps
Premier League +12.4% 0.0587 0.0671 11398
La Liga +22.2% 0.0555 0.0713 11080
Serie A +18.9% 0.0537 0.0663 10657
Bundesliga +13.5% 0.0619 0.0716 9039
Ligue 1 +18.9% 0.0612 0.0755 8894
Champions League +23.2% 0.0588 0.0765 3487
Detailed Decile Analysis

Players sorted by predicted scoring probability, split into 10 equal groups.

Decile Predicted Actual Avg Goals Apps
1 (Bottom) 8.1% 10.5% 0.113 5455
2 10.7% 11.3% 0.122 5455
3 12.2% 10.8% 0.118 5455
4 13.1% 9.3% 0.1 5455
5 14.0% 11.3% 0.126 5455
6 15.2% 11.5% 0.128 5455
7 16.5% 11.8% 0.129 5455
8 17.6% 12.8% 0.14 5455
9 19.9% 17.6% 0.201 5455
10 (Top) 28.6% 26.9% 0.323 5460
Performance by Season

SFM maintains strong performance across different seasons.

Season Impr. SFM Score Rate Apps
2025/26 +24.2% 0.0514 11.5% 35663
2024/25 +2.4% 0.0706 17.0% 18892