Model Validation

Model Performance Metrics and Diagnostics on Out-of-Sample Data

16.4%

Better than Player Historical Avg

2.55x

Top vs Bottom Decile Lift

48441

Player Appearances

0.152

Prediction Correlation

Methodology: All metrics use multi-class Brier score comparing full goal distributions (0, 1, 2, 3+ goals) on held-out test data (48441 player appearances). The Naive model uses each player's Poisson distribution derived from their historical scoring rate in the training data. Both SFM and Naive are evaluated on the same task: predicting the complete goal probability distribution, not just "score vs not score."

Calibration: Predicted vs Actual Scoring Rates

When we predict a 30% chance of scoring, does it happen ~30% of the time? Perfect calibration follows the diagonal line.

Ranking Performance: Scoring Rate by Prediction Decile

Players ranked in the top decile by our predictions score 27.3% of the time, compared to 10.7% for the bottom decile - a 2.55x lift.

Brier Score Comparison

Lower is better. Brier Score measures the accuracy of probabilistic predictions.

Model	Brier Score	vs SFM
SFM	0.0588	-
Naive	0.0704	+16.4%

Naive: Poisson distribution using each player's historical scoring rate from training data.

Based on 30072 player appearances with training history.

Performance by League

SFM consistently outperforms the Naive model across all 5 major European leagues.

League	Impr.	SFM	Naive	Apps
Premier League	+11.9%	0.0597	0.0677	10141
La Liga	+20.9%	0.0562	0.0711	9557
Serie A	+17.7%	0.0548	0.0666	9384
Bundesliga	+12.1%	0.0624	0.0709	8061
Ligue 1	+18.5%	0.062	0.0761	7955
Champions League	+23.4%	0.0585	0.0764	3343

Detailed Decile Analysis

Players sorted by predicted scoring probability, split into 10 equal groups.

Decile	Predicted	Actual	Avg Goals	Apps
1 (Bottom)	8.1%	10.7%	0.115	4844
2	10.7%	11.5%	0.124	4844
3	12.2%	11.0%	0.12	4844
4	13.1%	8.9%	0.095	4844
5	13.9%	11.7%	0.132	4844
6	15.2%	11.7%	0.129	4844
7	16.4%	12.2%	0.133	4844
8	17.5%	13.0%	0.143	4844
9	19.7%	18.1%	0.206	4844
10 (Top)	28.5%	27.3%	0.329	4845

Performance by Season

SFM maintains strong performance across different seasons.

Season	Impr.	SFM	Score Rate	Apps
2025/26	+24.3%	0.0512	11.5%	29549
2024/25	+2.4%	0.0707	17.0%	18892