Track record

Every night the model's newest prediction for each upcoming match is saved. The last one saved before a match kicks off is its locked prediction. Only locked predictions are scored, so nothing can be adjusted after the result. Backtests (predictions of past seasons) are never mixed in: they are on the methodology page.

So far

96 matches scored. Against base rates: model better (log loss difference −0.120 ± 0.073).

Locked predictions
97
Scored
96
Awaiting a result
0
Voided (postponed, suspended, cancelled)
1
Not counted: made after the actual kick-off
0
Missed: no prediction before kick-off
0
Awarded (decided off the pitch), not scored
0

Recording since . Matches before then have no locked predictions, and never count as missed. Updated .

How the model compares

Each scored match is also forecast by two others, on exactly the same matches: base rates (the league's usual home, draw and away frequencies, i.e. knowing nothing about the teams) and the market (bookmakers' closing odds with their margin removed, which arrive a few days after each match). The score is log loss: lower is better, and it punishes confident mistakes hardest.

The difference is the model's log loss minus the other's (negative = model better), with a 95% interval. If the interval includes zero, the verdict is “no detectable difference”.

Against the market: 96 matches so far, too few to judge: roughly 225 are needed to detect a gap the size the backtest found (0.02 in log loss).

Against base rates

LeagueMatchesModelBase ratesDifference ± 95% intervalFirst halfSecond halfVerdictMatches needed
All leagues960.9241.044−0.120 ± 0.073−0.104−0.136model better80
Premier League200.9391.037−0.098 ± 0.155+0.034−0.230no detectable difference76
Bundesliga181.0101.042−0.032 ± 0.160+0.019−0.082no detectable difference72
La Liga200.8991.111−0.213 ± 0.118−0.188−0.238model better44
Serie A200.9631.016−0.054 ± 0.200−0.191+0.084no detectable difference126
Ligue 1180.8051.011−0.206 ± 0.169−0.163−0.249model better81

Against the market

LeagueMatchesModelMarketDifference ± 95% intervalFirst halfSecond halfVerdictMatches needed
All leagues960.9240.913+0.010 ± 0.031+0.002+0.019no detectable difference225
Premier League200.9390.882+0.057 ± 0.057+0.057+0.058model worse165
Bundesliga181.0101.041−0.031 ± 0.068−0.034−0.028no detectable difference207
La Liga200.8990.870+0.029 ± 0.053+0.083−0.026no detectable difference143
Serie A200.9630.942+0.021 ± 0.087−0.047+0.090no detectable difference376
Ligue 1180.8050.837−0.033 ± 0.070−0.040−0.025no detectable difference222

“First half” and “second half” split the matches in date order. A difference that holds in only one half is “consistently better (or worse) but modest”. “Matches needed” is roughly how many matches it takes to detect a gap the size the backtest found: 0.08 in log loss against base rates, 0.02 against the market. Below that, “no detectable difference” means “not enough matches yet”, not “no better”.

Calibration

When the model says 60%, does it happen about 60% of the time? Forecasts are grouped by the chance they gave, and each group's average forecast is compared with how often the outcome happened.

Across: the average forecast in each group. Up: how often it happened, with its 95% interval. On the diagonal = honest forecasts.

Calibration, all leagues
OutcomeForecast rangeForecastsAverage forecastHappened95% interval
Home win0%–20%1017%20%6%–51%
Home win20%–40%3931%33%21%–49%
Home win40%–60%3150%61%44%–76%
Home win60%–80%1469%86%60%–96%
Home win80%–100%284%>99%34%–>99%
Draw0%–20%1116%27%10%–57%
Draw20%–40%8525%21%14%–31%
Away win0%–20%2214%5%1%–22%
Away win20%–40%4431%18%10%–32%
Away win40%–60%2649%62%43%–78%
Away win60%–80%466%50%15%–85%

Scored matches

The 50 most recent of 96, newest first. Times in UTC. Every scored match is in summary.json on the data branch.

Scored matches, newest first
Kick-offMatchScoreHomeDrawAwayLog lossPredictedModel version
Levante v Athletic BilbaoLa Liga3–137%27%36%1.000e1e4276
Getafe v Rayo VallecanoLa Liga1–137%35%28%1.047a31da68
Tottenham Hotspur v Coventry CityPremier League3–155%25%20%0.602a31da68
Parma v TorinoSerie A0–333%30%37%0.993a31da68
Monza v CagliariSerie A4–030%27%43%1.188a31da68
Real Madrid v SevillaLa Liga4–175%16%9%0.281a31da68
Lyon v NiceLigue 12–156%24%20%0.571a31da68
Juventus v LazioSerie A0–348%28%24%1.411a31da68
Málaga v Real SociedadLa Liga2–529%27%44%0.819a31da68
AC Milan v AtalantaSerie A3–240%27%33%0.915a31da68
Nottingham Forest v ArsenalPremier League0–118%27%55%0.595a31da68
Borussia Mönchengladbach v TSG HoffenheimBundesliga1–241%23%36%1.014a31da68
Lorient v MonacoLigue 11–030%25%45%1.193a31da68
Paris FC v RennesLigue 14–342%25%33%0.876a31da68
Le Havre v AuxerreLigue 11–337%29%34%1.091a31da68
Celta Vigo v AlavésLa Liga2–150%27%23%0.688a31da68
Bayer Leverkusen v SC FreiburgBundesliga2–162%20%18%0.476a31da68
Fiorentina v ComoSerie A1–325%26%49%0.716a31da68
Leeds United v Manchester UnitedPremier League1–136%27%37%1.304a31da68
Brighton & Hove Albion v Crystal PalacePremier League5–054%25%21%0.615a31da68
Bournemouth v SunderlandPremier League2–047%27%26%0.761a31da68
Angers v MarseilleLigue 12–520%25%55%0.589a31da68
Osasuna v Racing SantanderLa Liga2–059%21%20%0.524a31da68
Udinese v LecceSerie A0–350%28%22%1.527a31da68
Valencia v Athletic BilbaoLa Liga0–036%29%35%1.229de5189c
Troyes v LensLigue 11–316%21%63%0.456de5189c
Roma v GenoaSerie A2–065%22%13%0.433de5189c
Lille v BrestLigue 11–157%24%19%1.428de5189c
Real Betis v BarcelonaLa Liga2–524%22%54%0.612de5189c
Newcastle United v Aston VillaPremier League0–144%25%31%1.162de5189c
Bayern Munich v RB LeipzigBundesliga6–170%16%14%0.353de5189c
Bologna v Inter MilanSerie A1–319%22%59%0.530de5189c
Strasbourg v Paris Saint-GermainLigue 11–323%22%55%0.589de5189c
Villarreal v ElcheLa Liga0–067%18%15%1.719de5189c
Brentford v LiverpoolPremier League1–235%25%40%0.921de5189c
Manchester City v Ipswich TownPremier League3–286%10%4%0.158de5189c
Fulham v Hull CityPremier League1–048%28%24%0.732de5189c
Union Berlin v Borussia DortmundBundesliga0–017%23%60%1.474de5189c
Hamburger SV v VfB StuttgartBundesliga1–023%24%53%1.458de5189c
Werder Bremen v Paderborn 07Bundesliga2–249%28%23%1.277de5189c
Schalke 04 v Mainz 05Bundesliga0–130%29%41%0.882de5189c
SV Elversberg v FC AugsburgBundesliga4–339%28%33%0.947de5189c
Venezia v NapoliSerie A1–013%20%67%2.087de5189c
Espanyol v Atlético MadridLa Liga0–125%25%50%0.695de5189c
Everton v ChelseaPremier League0–234%28%38%0.976de5189c
Deportivo La Coruña v LevanteLa Liga1–143%24%33%1.420de5189c
Frosinone v SassuoloSerie A4–233%25%42%1.110de5189c
Le Mans v ToulouseLigue 10–333%26%41%0.884de5189c
Eintracht Frankfurt v 1. FC KölnBundesliga1–057%21%22%0.571de5189c
Levante v SevillaLa Liga3–345%25%30%1.3833796e2c

How the record is kept honest

  • The lock is the last prediction made before kick-off. “Made” means when it was written to disk, the strictest reading.
  • Postponed, suspended or cancelled: the lock is kept but voided, and never scored. A rearranged match gets a new lock before its new kick-off.
  • Checked against the real kick-off: a prediction that turns out to be from after the actual kick-off is not counted. A match no update predicted in time is recorded as missed.
  • Edits show: each line of the lock log carries a fingerprint (sha256) of the line before it, so changing or deleting an old line breaks the chain. The pipeline checked the chain before writing this summary.
  • Timing: every night's files are committed to the public data branch, which is protected against rewriting. That is strong evidence of when each prediction was published, not proof: it rests on GitHub's records.

Notes from the pipeline

  • Only predictions locked before kick-off by the daily run are scored. Backtests and any retrospective predictions are never mixed in.
  • Differences are model minus the other forecaster in log loss (negative = model better), with a 95% interval. With few matches the intervals are wide; matches_needed is roughly how many matches it would take to detect a gap the size the backtest found (0.08 against base rates, 0.02 against the market).
  • Market probabilities come from closing odds with the margin removed proportionally (an assumption). They appear a few days after each match.