Scoring rules reference

This page covers the multivariate ensemble scores, the weighted (threshold- and outcome-weighted) extensions, the censored and conditional likelihood scores, the quantile and interval scores, and the ranked probability score. All functions follow the lower-is-better convention.

For the three main univariate scores (crps, logs, dss) see Getting started and Forecast input modes.

Multivariate ensemble scores

These scores operate on a d × m matrix X where each column is one ensemble member, and a length-d observation vector y.

Energy score

where are independent draws from the forecast and   is the Euclidean norm. The sample approximation is

using ScoringRules, Distributions

d, m = 4, 500
X = randn(d, m)
y = randn(d)

es(X, y)

1.5385569816281013

# Per-member weights (normalised internally)
w = rand(m) .+ 0.1
es(X, y; w = w)

1.524354905511568

Variogram score

The variogram score of order penalises differences in the variogram structure between the forecast ensemble and the observation:

The default order is  . A   non-negative symmetric pairwise weight matrix can be supplied as w_vs, and per-member weights as w (as for es).

vs(X, y)            # default order p = 0.5

1.9814779836974905

vs(X, y; p = 1.0)   # order p = 1

9.37997809254588

w_vs = [1 / (1 + abs(k - l)) for k in 1:d, l in 1:d]
vs(X, y; w = w, w_vs = w_vs)

0.863772329868199

MMD score

The maximum-mean-discrepancy score uses a Gaussian kernel    :

mmds(X, y)

-0.013245787256084246

mmds(X, y; w = w)   # per-member weights

-0.013029638171145316

Weighted scores

Weighted scoring rules let you emphasise a particular region of the outcome space. Two weighting mechanisms are available for each score.

Threshold-weighted (tw*) scores apply a chaining function to the observations and ensemble members before computing the standard score. The default chaining function is  , which focuses attention on outcomes in the interval .

Outcome-weighted (ow*) scores instead assign each ensemble member a weight function value . The default weight function is the indicator    .

Threshold- and outcome-weighted CRPS

dat = randn(500)
y1d = 0.5

# Score outcomes above 1.0 more heavily
twcrps(dat, y1d; a = 1.0)

0.007067953510274917

owcrps(dat, y1d; a = 1.0)

0.0

# Custom chaining function
twcrps(dat, y1d; chain_func = z -> max(z, 0.0))

0.21703948864485637

Censored and conditional likelihood scores

clogs computes the censored (cens = true, the default) and conditional (cens = false) likelihood scores of Diks, Panchenko & van Dijk (2011), based on a Gaussian kernel density estimate of the ensemble and the window , with    the estimated probability of the window.

The censored score treats all outcomes outside the window as a single event: it is inside the window and   outside. The conditional score is inside the window and outside.

clogs(dat, y1d; a = 1.0)                 # censored likelihood score

0.16977610938499857

clogs(dat, y1d; a = 1.0, cens = false)   # conditional likelihood score

0.0

With the default unbounded window, clogs reduces to the KDE-based logs; the bandwidth can be set via bw (Silverman's rule-of-thumb by default).

Threshold- and outcome-weighted energy score

# Focus on outcomes above 0 in all dimensions
twes(X, y; a = 0.0)

0.7809793115632303

owes(X, y; a = 0.0)

0.0

Threshold- and outcome-weighted variogram score

twvs(X, y; a = 0.0)

1.6036138483825555

owvs(X, y; a = 0.0)

0.0

Threshold- and outcome-weighted MMD score

twmmds(X, y; a = 0.0)

-0.1544483286285489

owmmds(X, y; a = 0.0)

0.0

For multivariate weighted scores, a and b can be scalars (broadcast to all dimensions) or length-d vectors. A custom chain_func or weight_func can be supplied when the default interval-based functions do not suit. All weighted scores also take per-member weights w: the tw* scores weight the chained ensemble, and the ow* scores multiply the outcome weights by the member weights, as in R.

Quantile and interval scores

Quantile score (pinball loss)

The quantile score at level with forecast and observation is

which equals    when   and   when  .

levels = [0.1, 0.5, 0.9]
qs     = [-1.28, 0.0, 1.28]  # standard-normal quantiles
quantile_score(levels, qs, 1.0)

3-element Vector{Float64}:
 0.22800000000000004
 0.5
 0.027999999999999997

mean(quantile_score(levels, qs, 1.0))   # mean quantile score

0.252

Interval score

The interval score for a   central prediction interval is

In this package the level argument is the nominal coverage   (e.g. level = 0.9 for a 90% interval):

# 90% prediction interval for N(0,1): roughly (−1.645, 1.645)
interval_score(-1.645, 1.645, 0.5, 0.9)

3.2900000000000014

Ranked probability score

The ranked probability score (Epstein 1969) is for categorical / ordinal forecasts over   ordered categories. Given a vector of forecast probabilities   and an observed category  :

p = [0.2, 0.5, 0.3]   # probabilities over 3 ordered categories
rps(p, 2)              # category 2 was observed

0.13000000000000003

rps(p, 1)

0.7300000000000002

References

• Jordan, A., Krüger, F., & Lerch, S. (2019). Evaluating Probabilistic Forecasts with scoringRules. Journal of Statistical Software, 90(12), 1–37. doi:10.18637/jss.v090.i12

• Allen, S. (2024). Weighted scoringRules: Emphasizing Particular Outcomes When Evaluating Probabilistic Forecasts. Journal of Statistical Software, 110(8), 1–26. doi:10.18637/jss.v110.i08

• Diks, C., Panchenko, V., & van Dijk, D. (2011). Likelihood-based scoring rules for comparing density forecasts in tails. Journal of Econometrics, 163, 215–230. doi:10.1016/j.jeconom.2011.04.001

• Gneiting, T. & Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. JASA, 102, 359–378.

• Scheuerer, M. & Hamill, T. M. (2015). Variogram-based proper scoring rules for probabilistic forecasts of multivariate quantities. Monthly Weather Review, 143, 1321–1334.

• Epstein, E. S. (1969). A scoring system for probability forecasts of ranked categories. Journal of Applied Meteorology and Climatology, 8, 985–987.