Track record
Every night the model's newest prediction for each upcoming match is saved. The last one saved before a match kicks off is its locked prediction. Only locked predictions are scored, so nothing can be adjusted after the result. Backtests (predictions of past seasons) are never mixed in: they are on the methodology page.
So far
250 matches scored. Compared with league-average forecasts: model better. Compared with bookmakers' odds: no detectable difference.
- Locked predictions
- 251
- Scored
- 250
- Awaiting a result
- 0
- Voided (postponed, suspended, cancelled)
- 1
- Not counted: made after the actual kick-off
- 0
- Missed: no prediction before kick-off
- 0
- Awarded (decided off the pitch), not scored
- 0
Recording since . Matches before then have no locked predictions, and never count as missed. Updated .
How the model compares
Each scored match is also forecast by two others, on exactly the same matches: base rates (the league's usual home, draw and away frequencies, i.e. knowing nothing about the teams) and the market (bookmakers' closing odds with their margin removed, which arrive a few days after each match). The score is log loss: lower is better, and it punishes confident mistakes hardest.
The difference is the model's log loss minus the other's (negative = model better), with a 95% interval. If the interval includes zero, the verdict is “no detectable difference”.
Against base rates
| League | Matches | Model | Base rates | Difference ± 95% interval | First half | Second half | Verdict | Matches needed |
|---|---|---|---|---|---|---|---|---|
| All leagues | 250 | 0.929 | 1.050 | −0.121 ± 0.044 | −0.113 | −0.128 | model better | 77 |
| Premier League | 50 | 1.003 | 1.049 | −0.046 ± 0.096 | −0.069 | −0.024 | no detectable difference | 73 |
| Bundesliga | 45 | 0.954 | 1.046 | −0.092 ± 0.104 | 0.000 | −0.180 | no detectable difference | 77 |
| La Liga | 50 | 0.926 | 1.087 | −0.160 ± 0.087 | −0.188 | −0.132 | model better | 60 |
| Serie A | 60 | 0.895 | 1.042 | −0.147 ± 0.105 | −0.129 | −0.165 | model better | 104 |
| Ligue 1 | 45 | 0.871 | 1.024 | −0.153 ± 0.094 | −0.172 | −0.135 | model better | 63 |
Against the market
| League | Matches | Model | Market | Difference ± 95% interval | First half | Second half | Verdict | Matches needed |
|---|---|---|---|---|---|---|---|---|
| All leagues | 250 | 0.929 | 0.915 | +0.015 ± 0.016 | +0.005 | +0.024 | no detectable difference | 163 |
| Premier League | 50 | 1.003 | 0.967 | +0.036 ± 0.032 | +0.054 | +0.018 | model worse | 130 |
| Bundesliga | 45 | 0.954 | 0.980 | −0.026 ± 0.035 | −0.041 | −0.012 | no detectable difference | 139 |
| La Liga | 50 | 0.926 | 0.874 | +0.052 ± 0.034 | +0.029 | +0.076 | model worse | 148 |
| Serie A | 60 | 0.895 | 0.893 | +0.002 ± 0.037 | +0.013 | −0.008 | no detectable difference | 202 |
| Ligue 1 | 45 | 0.871 | 0.865 | +0.006 ± 0.038 | −0.030 | +0.040 | no detectable difference | 162 |
“First half” and “second half” split the matches in date order. A difference that holds in only one half is “consistently better (or worse) but modest”. “Matches needed” is roughly how many matches it takes to detect a gap the size the backtest found: 0.08 in log loss against base rates, 0.02 against the market. Below that, “no detectable difference” means “not enough matches yet”, not “no better”.
Calibration
When the model says 60%, does it happen about 60% of the time? Forecasts are grouped by the chance they gave, and each group's average forecast is compared with how often the outcome happened.
Across: the average forecast in each group. Up: how often it happened, with its 95% interval. On the diagonal = honest forecasts.
| Outcome | Forecast range | Forecasts | Average forecast | Happened | 95% interval |
|---|---|---|---|---|---|
| Home win | 0%–20% | 21 | 16% | 14% | 5%–35% |
| Home win | 20%–40% | 95 | 31% | 35% | 26%–45% |
| Home win | 40%–60% | 91 | 49% | 55% | 45%–65% |
| Home win | 60%–80% | 39 | 67% | 77% | 62%–87% |
| Home win | 80%–100% | 4 | 84% | >99% | 51%–>99% |
| Draw | 0%–20% | 32 | 17% | 16% | 7%–32% |
| Draw | 20%–40% | 218 | 25% | 22% | 17%–27% |
| Away win | 0%–20% | 56 | 14% | 9% | 4%–19% |
| Away win | 20%–40% | 118 | 30% | 22% | 16%–30% |
| Away win | 40%–60% | 65 | 48% | 58% | 46%–70% |
| Away win | 60%–80% | 11 | 68% | 82% | 52%–95% |
Scored matches
The 50 most recent of 250, newest first. Times in UTC. Every scored match is in summary.json on the data branch.
| Kick-off | Match | Score | Home | Draw | Away | Log loss | Predicted | Model version |
|---|---|---|---|---|---|---|---|---|
| Atlético Madrid v BarcelonaLa Liga | 1–2 | 34% | 24% | 42% | 0.867 | 8c2a856 | ||
| Lens v MarseilleLigue 1 | 1–2 | 46% | 24% | 30% | 1.211 | 8c2a856 | ||
| Fiorentina v JuventusSerie A | 2–2 | 26% | 27% | 47% | 1.330 | 8c2a856 | ||
| TSG Hoffenheim v Schalke 04Bundesliga | 2–1 | 58% | 24% | 18% | 0.544 | 8c2a856 | ||
| Celta Vigo v LevanteLa Liga | 0–0 | 53% | 24% | 23% | 1.432 | 8c2a856 | ||
| Villarreal v GetafeLa Liga | 2–0 | 59% | 25% | 16% | 0.520 | 8c2a856 | ||
| Inter Milan v ComoSerie A | 2–0 | 50% | 24% | 26% | 0.703 | 8c2a856 | ||
| Manchester United v Aston VillaPremier League | 4–4 | 42% | 26% | 32% | 1.349 | 8c2a856 | ||
| SC Freiburg v Union BerlinBundesliga | 2–0 | 57% | 24% | 19% | 0.572 | 8c2a856 | ||
| Le Mans v MonacoLigue 1 | 1–1 | 21% | 22% | 57% | 1.503 | 8c2a856 | ||
| Brest v LyonLigue 1 | 2–0 | 32% | 25% | 43% | 1.146 | 8c2a856 | ||
| Le Havre v LorientLigue 1 | 0–2 | 33% | 29% | 38% | 0.969 | 8c2a856 | ||
| Angers v NiceLigue 1 | 0–2 | 27% | 29% | 44% | 0.815 | 8c2a856 | ||
| Real Sociedad v Rayo VallecanoLa Liga | 1–0 | 50% | 26% | 24% | 0.689 | 8c2a856 | ||
| 1. FC Köln v Bayer LeverkusenBundesliga | 1–2 | 23% | 23% | 54% | 0.611 | 8c2a856 | ||
| Monza v AtalantaSerie A | 0–1 | 19% | 20% | 61% | 0.497 | 8c2a856 | ||
| Sunderland v ChelseaPremier League | 1–0 | 28% | 27% | 45% | 1.282 | 8c2a856 | ||
| Genoa v AC MilanSerie A | 1–2 | 23% | 26% | 51% | 0.672 | 8c2a856 | ||
| Rennes v LilleLigue 1 | 1–2 | 36% | 27% | 37% | 0.981 | 8c2a856 | ||
| Crystal Palace v LiverpoolPremier League | 1–3 | 23% | 25% | 52% | 0.650 | 8c2a856 | ||
| Ipswich Town v BournemouthPremier League | 1–2 | 24% | 24% | 52% | 0.647 | 8c2a856 | ||
| Brighton & Hove Albion v BrentfordPremier League | 2–0 | 48% | 25% | 27% | 0.725 | 8c2a856 | ||
| Elche v Real BetisLa Liga | 0–2 | 35% | 26% | 39% | 0.942 | 8c2a856 | ||
| Napoli v LazioSerie A | 1–1 | 45% | 27% | 28% | 1.301 | 8c2a856 | ||
| Paris Saint-Germain v TroyesLigue 1 | 1–0 | 87% | 9% | 4% | 0.140 | 8c2a856 | ||
| Valencia v Real MadridLa Liga | 0–3 | 17% | 23% | 60% | 0.515 | 8c2a856 | ||
| Roma v SassuoloSerie A | 3–0 | 71% | 17% | 12% | 0.348 | 8c2a856 | ||
| Auxerre v Paris FCLigue 1 | 0–1 | 30% | 27% | 43% | 0.832 | 8c2a856 | ||
| VfB Stuttgart v Werder BremenBundesliga | 1–0 | 62% | 20% | 18% | 0.481 | 8c2a856 | ||
| Nottingham Forest v Manchester CityPremier League | 3–1 | 19% | 25% | 56% | 1.657 | 8c2a856 | ||
| Osasuna v Athletic BilbaoLa Liga | 1–1 | 34% | 29% | 37% | 1.245 | 8c2a856 | ||
| Parma v BolognaSerie A | 1–2 | 25% | 29% | 46% | 0.769 | 8c2a856 | ||
| Espanyol v Deportivo La CoruñaLa Liga | 0–0 | 49% | 27% | 24% | 1.326 | 8c2a856 | ||
| Fulham v Newcastle UnitedPremier League | 1–1 | 37% | 27% | 36% | 1.306 | 8c2a856 | ||
| Arsenal v Hull CityPremier League | 1–0 | 75% | 18% | 7% | 0.284 | 8c2a856 | ||
| RB Leipzig v FC AugsburgBundesliga | 5–2 | 66% | 19% | 15% | 0.420 | 8c2a856 | ||
| Borussia Dortmund v SV ElversbergBundesliga | 2–1 | 72% | 17% | 11% | 0.326 | 8c2a856 | ||
| Paderborn 07 v Eintracht FrankfurtBundesliga | 2–2 | 32% | 28% | 40% | 1.283 | 8c2a856 | ||
| Mainz 05 v Bayern MunichBundesliga | 1–2 | 13% | 17% | 70% | 0.358 | 8c2a856 | ||
| Torino v LecceSerie A | 1–1 | 49% | 28% | 23% | 1.276 | 8c2a856 | ||
| Cagliari v FrosinoneSerie A | 1–0 | 35% | 25% | 40% | 1.051 | 8c2a856 | ||
| Málaga v Racing SantanderLa Liga | 1–2 | 47% | 24% | 29% | 1.255 | 8c2a856 | ||
| Leeds United v Tottenham HotspurPremier League | 1–0 | 41% | 27% | 32% | 0.885 | 8c2a856 | ||
| Sevilla v AlavésLa Liga | 0–0 | 48% | 26% | 26% | 1.344 | 8c2a856 | ||
| Everton v Coventry CityPremier League | 3–1 | 53% | 27% | 20% | 0.645 | 8c2a856 | ||
| Venezia v UdineseSerie A | 2–1 | 32% | 27% | 41% | 1.152 | 8c2a856 | ||
| Toulouse v StrasbourgLigue 1 | 3–1 | 39% | 26% | 35% | 0.928 | 8c2a856 | ||
| Hamburger SV v Borussia MönchengladbachBundesliga | 3–1 | 38% | 27% | 35% | 0.971 | 8c2a856 | ||
| Deportivo La Coruña v OsasunaLa Liga | 2–1 | 41% | 27% | 32% | 0.888 | 8c2a856 | ||
| Newcastle United v EvertonPremier League | 0–1 | 48% | 27% | 25% | 1.376 | 8c2a856 |
How the record is kept honest
- The lock is the last prediction made before kick-off. “Made” means when it was written to disk, the strictest reading.
- Postponed, suspended or cancelled: the lock is kept but voided, and never scored. A rearranged match gets a new lock before its new kick-off.
- Checked against the real kick-off: a prediction that turns out to be from after the actual kick-off is not counted. A match no update predicted in time is recorded as missed.
- Edits show: each line of the lock log carries a fingerprint (sha256) of the line before it, so changing or deleting an old line breaks the chain. The pipeline checked the chain before writing this summary.
- Timing: every night's files are committed to the public data branch, which is protected against rewriting. That is strong evidence of when each prediction was published, not proof: it rests on GitHub's records.
Notes from the pipeline
- Only predictions locked before kick-off by the daily run are scored. Backtests and any retrospective predictions are never mixed in.
- Differences are the model's log loss minus the other forecaster's (negative = model better), with a 95% interval. With few matches the intervals are wide. Matches needed is roughly how many matches it would take to detect a gap the size the backtest found (0.08 in log loss against base rates, 0.02 against the market).
- Market probabilities come from closing odds with the margin removed proportionally (an assumption). They appear a few days after each match.