s10479-022-04611-9.pdf (2.42 MB)
Evaluating the discrimination ability of proper multi-variate scoring rules
Version 2 2023-06-12, 07:43
Version 1 2023-06-10, 02:50
journal contribution
posted on 2023-06-12, 07:43 authored by Carol AlexanderCarol Alexander, Michael Coulon, Y Han, Xiaochun MengXiaochun MengProper scoring rules are commonly applied to quantify the accuracy of distribution forecasts. Given an observation they assign a scalar score to each distribution forecast, with the lowest expected score attributed to the true distribution. The energy and variogram scores are two rules that have recently gained some popularity in multivariate settings because their computation does not require a forecast to have parametric density function and so they are broadly applicable. Here we conduct a simulation study to compare the discrimination ability between the energy score and three variogram scores. Compared with other studies, our simulation design is more realistic because it is supported by a historical data set containing commodity prices, currencies and interest rates, and our data generating processes include a diverse selection of models with different marginal distributions, dependence structure, and calibration windows. This facilitates a comprehensive comparison of the performance of proper scoring rules in different settings. To compare the scores we use three metrics: the mean relative score, error rate and a generalized discrimination heuristic. Overall, we find that the variogram score with parameter p=0.5 outperforms the energy score and the other two variogram scores.
History
Publication status
- Published
File Version
- Published version
Journal
Annals of Operations ResearchISSN
0254-5330Publisher
SpringerExternal DOI
Department affiliated with
- Accounting and Finance Publications
Full text available
- Yes
Peer reviewed?
- Yes
Legacy Posted Date
2022-03-07First Open Access (FOA) Date
2022-03-18First Compliant Deposit (FCD) Date
2022-03-05Usage metrics
Categories
No categories selectedKeywords
Licence
Exports
RefWorks
BibTeX
Ref. manager
Endnote
DataCite
NLM
DC