Scoring

Every number below is executed before it is published. The worked examples on this page are the same values the test suite asserts against, so the page and the code cannot drift apart.

The chain, in order

1. Each appearance is converted to the band it falls in, against the frozen reference pool.
2. Those bands form the case's realised distribution, rebuilt from full history every week.
3. The prediction is scored against it with the ranked probability score.
4. Scores are averaged over metrics, then over cases.
5. The result is divided by what the league base rate scored on the same outcomes.

Realised distribution

For one case and one metric, the histogram of every band observed so far.

A striker with eight appearances — five scoreless, two with one goal, one with two — on the three-band grid 0 / 1 / 2+ gives:

ĝ = [0.625, 0.250, 0.125]

Rebuilt in full each week, never accumulated. That is why the per-appearance band history is stored rather than a running score: adding week nine has to be able to change what weeks one to eight measured, and a running total could not.

Ranked probability score

The distance between the predicted distribution and the realised one, compared through their cumulative distributions:

RPS(p, g) = 1/(K-1) · Σ (F_k − G_k)²      over k = 1 … K−1

where F and G are the cumulative sums of the predicted and realised distributions, and K is the number of bands. Bounded in [0, 1], lower is better. The K−1 divisor is what makes a three-band metric and a ten-band metric comparable, so they can be averaged directly.

Two properties matter:

• Order-aware. Being one band out costs less than being five bands out. A squared-error on cumulative mass carries the ordering of the bands, which a plain histogram distance would throw away.
• Proper. A participant minimises its expected score by reporting what it actually believes. There is no hedge, no shading, and no confidence level that improves the expected result.

Worked example

Prediction p = [0.70, 0.25, 0.05], so F = [0.70, 0.95]. Realised ĝ = [0.625, 0.250, 0.125], so G = [0.625, 0.875].

RPS = ½ · [ (0.70 − 0.625)² + (0.95 − 0.875)² ] = 0.00562

Earlier in the season, when only the first two scoreless appearances had been played, the same prediction scored 0.04625 against ĝ = [1, 0, 0]. The score is recomputed from scratch each week and is not carried forward.

Why not average per-appearance scores

Scoring each appearance separately and averaging gives a strictly larger number, and the difference is exactly identifiable:

mean over appearances = RPS(p, ĝ) + 1/(K−1) · Σ G_k (1 − G_k)

The extra term is the dispersion of the outcomes. It contains no prediction, it is identical for every participant on that case, and no model can reduce it. Including it would add a constant to everyone's score that varies by case — noise in the ranking, dressed as measurement. Verified over 2,000 random cases.

Aggregation

Two unweighted means, in order: over the metrics in the case's mask, then over the participant's scored cases.

Cases are weighted equally regardless of appearances. An ever-present signing does not outweigh the rest of the cohort — one transfer is one unit of evidence about a model's ability to project transfers, however often that player happens to play.

Only scored cases enter. Refused, unobserved and void cases never do.

Skill score

The baseline is the pooled distribution — the league base rate for that competition, role and metric, read from the frozen reference pool and run through exactly the same steps on exactly the same realised distributions.

skill score = 1 − mean RPS ÷ mean RPS of the base rate

Continuing the example: the Premier League base rate for a striker's goals is [0.78, 0.19, 0.03], which scores 0.01653 against the same outcomes. So

skill score = 1 − 0.00562 ÷ 0.01653 = 0.660

Zero means no better than knowing the league base rate for the role — that is, knowing nothing about the individual player. One is perfect. Negative values are published as-is: below zero is worse than a table of league averages, and a confidently wrong model belongs below a shrug.

The baseline is reachable by anyone

Bands are cut at cumulative mass, so a band's percentile width is its share of peer appearances. The base rate is therefore readable straight off the grid sent in the request. No privileged access to Arena data is required to score zero, which is what makes zero a fair floor rather than an insider's number.

Why not a uniform baseline

Uniform is only the know-nothing forecast where the pool is itself uniform across bands. For the count-aligned grids it is not: goals pools at roughly 78/19/3.

On the same example, a uniform baseline scores 0.06424, making the skill score 0.912 instead of 0.660 — a far easier bar. Measured on real Premier League players, a participant predicting nothing but the base rate scored 0.951 on ST/goals against uniform and 0.000 against the base rate. On CM/total_passes, where the pool is near-uniform, both give 0.000.

The flaw was invisible until the count-aligned grids arrived, which is a reason to publish the baseline choice rather than to leave it implied.

Computing coverage

coverage = scored cases ÷ scorable cases

The denominator is cohort-level: candidates minus void minus unobserved, identical for every participant. Refusals and never-predicted cases both reduce the numerator.

Published beside the skill score, never folded into it, so declining is visible and costed without a refusal corrupting the accuracy measure.

Confidence discrimination

Kendall's τ_b between a participant's stated confidence and its realised error, with the error negated so that positive means higher confidence went with lower error.

τ_b = (C − D) ÷ √((n₀ − n₁)(n₀ − n₂))

Take every pair of the participant's scored cases. A pair is concordant (C) when the case it was more confident about was also the one it got more right, and discordant (D) when the opposite. n₀ is the number of pairs; n₁ and n₂ subtract the pairs tied on confidence and on error respectively, since a tie carries no ordering information either way.

The result runs from −1 to +1. Published as its own ranked column, never folded into the skill score.

A flat confidence scores no data, not a high score. If every case carries the same value, n₀ − n₁ is zero and the measure is undefined. That is what makes the column unfarmable by assertion. And deliberately predicting badly on low-confidence cases would raise discrimination while lowering the skill score, which sits next to it.

A participant that omits confidence entirely shows no data, never zero, and its accuracy ranking is untouched.

Intervals

A bootstrap over cases: would this score be different on another cohort of transfers? Cases are resampled with replacement 1,000 times, and the interval is the 2.5th to 97.5th percentile of the resulting skill scores.

Three requirements, each of which changes the answer if broken:

1. Resample cases, not appearances. Appearances within a case share a player and are not independent evidence. One signing is one unit however often he plays.
2. One index set per draw, shared by numerator and denominator. The skill score is a ratio; drawing separately would compare a participant on one cohort against a baseline on another.
3. Seed from the run id, so a published interval is reproducible from stored artifacts.

Do not read significance from overlapping intervals

Participants face identical cases, so their errors are correlated and the difference between two scores is far better determined than either score alone.

On a worked twelve-case cohort: A scores 0.498 [0.387, 0.594] and B scores 0.454 [0.334, 0.559] — heavily overlapping. The paired difference is 0.044 [0.035, 0.053], which excludes zero and is twelve times tighter. A is ahead of B, and comparing the two published intervals would have said otherwise.

What the interval does not cover

With few appearances the realised distribution is coarse, and every participant is pulled toward the baseline. That compression is systematic, shared by everyone, and shrinks as appearances accumulate. It is not sampling noise over cases, and this interval does not measure it. It is carried instead by the appearance count and the provisional flag.

Edge cases

Determinism

Every published number is reproducible from stored artifacts alone. A weekly run pins the case registry, reference pool, entity crosswalk and scoring metric versions; bootstrap draws use a seed derived from the run id; and scores are appended rather than recomputed.

A number published in week 3 reads the same in week 30, and the rules that produced it are still named and readable.