Model Validation

Model Performance Metrics and Diagnostics on Out-of-Sample Data

17.3%

Better than Player Historical Avg

2.56x

Top vs Bottom Decile Lift

54574

Player Appearances

0.152

Prediction Correlation

Methodology: All metrics use multi-class Brier score comparing full goal distributions (0, 1, 2, 3+ goals) on held-out test data (54574 player appearances). The Naive model uses each player's Poisson distribution derived from their historical scoring rate in the training data. Both SFM and Naive are evaluated on the same task: predicting the complete goal probability distribution, not just "score vs not score."

Calibration: Predicted vs Actual Scoring Rates

When we predict a 30% chance of scoring, does it happen ~30% of the time? Perfect calibration follows the diagonal line.

Ranking Performance: Scoring Rate by Prediction Decile

Players ranked in the top decile by our predictions score 26.9% of the time, compared to 10.5% for the bottom decile - a 2.56x lift.

Brier Score Comparison

Lower is better. Brier Score measures the accuracy of probabilistic predictions.

Model	Brier Score	vs SFM
SFM	0.058	-
Naive	0.0702	+17.3%

Naive: Poisson distribution using each player's historical scoring rate from training data.

Based on 32738 player appearances with training history.

Performance by League

SFM consistently outperforms the Naive model across all 5 major European leagues.

League	Impr.	SFM	Naive	Apps
Premier League	+12.4%	0.0587	0.0671	11398
La Liga	+22.2%	0.0555	0.0713	11080
Serie A	+18.9%	0.0537	0.0663	10657
Bundesliga	+13.5%	0.0619	0.0716	9039
Ligue 1	+18.9%	0.0612	0.0755	8894
Champions League	+23.1%	0.0586	0.0762	3506

Detailed Decile Analysis

Players sorted by predicted scoring probability, split into 10 equal groups.

Decile	Predicted	Actual	Avg Goals	Apps
1 (Bottom)	8.1%	10.5%	0.113	5457
2	10.7%	11.3%	0.122	5457
3	12.2%	10.8%	0.118	5457
4	13.1%	9.2%	0.1	5457
5	14.0%	11.3%	0.126	5457
6	15.2%	11.6%	0.127	5457
7	16.5%	11.8%	0.129	5457
8	17.6%	12.8%	0.14	5457
9	19.9%	17.6%	0.201	5457
10 (Top)	28.6%	26.9%	0.322	5461

Performance by Season

SFM maintains strong performance across different seasons.

Season	Impr.	SFM	Score Rate	Apps
2025/26	+24.2%	0.0513	11.5%	35682
2024/25	+2.4%	0.0706	17.0%	18892