ScoringRules.jl

Appearance

1

ScoringRules ​

Proper scoring rules for evaluating probabilistic forecasts in Julia.

A port of the R scoringRules package ​

ScoringRules.jl is a port of the R package scoringRules by Alexander I. Jordan, Fabian Krüger, Sebastian Lerch and Sam Allen. Its formulae and algorithms follow that package, whose methods are rooted in the scientific literature on probabilistic forecasting. The package is expected to diverge from upstream over time. If you use it, please cite the original authors (see Attribution and provenance). It is distributed under the same licence as the original, GPL-2.0-or-later.

The initial port was generated by a large language model (Claude Opus 4.8) under human guidance, translating the R and C++ source into idiomatic Julia. It introduced no statistical methodology of its own.

The formulae and algorithms follow the R package; the interface is Julia's. What that means in practice:

  • Forecasts are Distributions.jl objects, dispatched on their type, instead of a separate function per family: write crps(Normal(0, 1), 0.4), not crps_norm.

  • Gradients come from automatic differentiation with any supported backend, so there is no equivalent of R's hand-written gradcrps_* / hesscrps_*.

  • Results match R throughout (the test suite checks against reference values generated from R; see Validation against R for a live value- and gradient-level comparison). The one genuine numerical difference is the log-logistic Dawid–Sebastiani score, where R has a bug that returns NaN and we return the correct value; a couple of other choices differ in method but not in results (e.g. an analytic GEV Gumbel limit in place of R's numerical integration). Both are documented under Differences from R.

  • LogLaplace, TwoPieceNormal and TwoPieceExponential are provided here, as Distributions.jl does not have them.

Overview ​

  • Univariate scores — CRPS (crps), the logarithmic score (logs) and the Dawid–Sebastiani score (dss), for parametric (Distributions.jl) and ensemble forecasts alike.

  • Multivariate scores — energy score (es), variogram score (vs) and the maximum-mean-discrepancy score (mmds) for ensemble forecasts.

  • Quantile and categorical scores — quantile_score, interval_score and the ranked probability score (rps).

All scores are negatively oriented, so a lower score is a better forecast, and they broadcast with the usual dot syntax.

Getting started ​

julia
using ScoringRules, Distributions

crps(Normal(0, 1), 0.4)          # parametric forecast
logs(Poisson(3), 2)              # log score of a count forecast
crps(randn(1000), 0.4)           # ensemble (sample) forecast
0.3017968895969185

See the documentation for the full list of supported families and scores.

  • scoringRules (R) — the reference implementation this package ports; the source of the parametric, sample and weighted scoring rules provided here, and of the values the test suite validates against.

  • scoringutils (R) — applied forecast-evaluation utilities built on top of scoringRules.

  • Distributions.jl — supplies the distribution types that the parametric scores dispatch on.

  • CensoredDistributions.jl — sibling EpiAware package whose structure, testing and CI this package follows.

  • PostForecasts.jl and ForecastBaselines.jl — other Julia packages that touch probabilistic-forecast scoring.

Where to learn more ​

Attribution and provenance ​

ScoringRules.jl began as a Julia port of the R scoringRules package (version 1.1.3), whose methods are rooted in the scientific literature on probabilistic forecasting. That package was written by:

Please cite their work:

Jordan, A., Krüger, F., & Lerch, S. (2019). Evaluating Probabilistic Forecasts with scoringRules. Journal of Statistical Software, 90(12), 1–37. https://doi.org/10.18637/jss.v090.i12

For the weighted scores (twcrps, owcrps, and relatives):

Allen, S. (2024). Weighted scoringRules: Emphasizing Particular Outcomes When Evaluating Probabilistic Forecasts. Journal of Statistical Software, 110(8), 1–26. https://doi.org/10.18637/jss.v110.i08

The initial LLM-generated port introduced no statistical methodology of its own. The package will grow features the R package does not have; the citation request above covers everything that traces back to scoringRules. Any errors from translation or later development are ours, not the original authors'; please report discrepancies with the R package as bugs here.

Contributing ​

We welcome contributions and new contributors! Please open an issue or pull request on GitHub. This package follows ColPrac and the SciML style.

How to cite ​

If you use ScoringRules in your work, please cite it. Citation metadata lives in CITATION.cff, which GitHub renders as a "Cite this repository" button on the repository page.

Code of conduct ​

Please note that the ScoringRules project is released with a Contributor Code of Conduct. By contributing, you agree to abide by its terms.