Fair scores for ensemble forecasts

Quarterly Journal of the Royal Meteorological Society

Volumn 140, Issue 683, 2014, Pages 1917-1923

  Ferro, C A T ^a  

^a UNIVERSITY OF EXETER   (United Kingdom)

Author keywords

Brier score; Continuous ranked probability score; Forecast verification; Scoring rules

Indexed keywords

EARTH ATMOSPHERE; METEOROLOGY;

BRIER SCORE; CONTINUOUS RANKED PROBABILITY SCORES; ENSEMBLE FORECASTS; FORECAST VERIFICATIONS; GENERAL CLASS; MULTICATEGORY; RANDOM SAMPLE; SCORING RULES;

FORECASTING;

EID: 84908878668     PISSN: 00359009     EISSN: 1477870X     Source Type: Journal    
DOI: 10.1002/qj.2270     Document Type: Article

References (29)

1. Anderson JL. 1996. A method for producing and evaluating probabilistic forecasts from ensemble model integrations. J. Clim. 9: 1518-1530, doi: 10.1175/1520-0442(1996)009<1518:AMFPAE>2.0.CO;2.
2. Brier GW. 1950. Verification of forecasts expressed in terms of probability. Mon. Weather Rev. 78: 1-3, doi: 10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2.
3. Bröcker J. 2012. Evaluating raw ensembles with the continuous ranked probability score. Q. J. R. Meteorol. Soc. 138: 1611-1617, doi: 10.1002/qj.1891.
4. Bröcker J, Kantz H. 2011. The concept of exchangeability in ensemble forecasting. Nonlinear Processes Geophys. 18: 1-5, doi: 10.5194/npg-18-1-2011.
5. Bröcker J, Smith LA. 2007. Scoring probabilistic forecasts: The importance of being proper. Weather and Forecasting 22: 382-388, doi: 10.1175/WAF966.1.
6. Brown TA. 1974. 'Admissible scoring systems for continuous distributions', Technical Note P-5235. The Rand Corporation: Santa Monica, CA.
7. Ferro CAT. 2007. Comparing probabilistic forecasting systems with the Brier score. Weather and Forecasting 22: 1076-1088, doi: 10.1175/WAF1034.1.
8. Ferro CAT, Stephenson DB. 2011. Extremal dependence indices: Improved verification measures for deterministic forecasts of rare binary events. Weather and Forecasting 26: 699-713, doi: 10.1175/WAF-D-10-05030.1.
9. Ferro CAT, Richardson DS, Weigel AP. 2008. On the effect of ensemble size on the discrete and continuous ranked probability scores. Meteorol. Appl. 15: 19-24, doi: 10.1002/met.45.
10. Fraley C, Raftery AE, Gneiting T. 2010. Calibrating multimodel forecast ensembles with exchangeable and missing members using Bayesian model averaging. Mon. Weather Rev. 138: 190-202, doi: 10.1175/2009MWR3046.1.
11. Fricker TE, Ferro CAT, Stephenson DB. 2013. Three recommendations for evaluating climate predictions. Meteorol. Appl. 20: 246-255, doi: 10.1002/met.1409.
12. Gneiting T. 2011. Making and evaluating point forecasts. J. Amer. Stat. Assoc. 106: 746-762, doi: 10.1198/jasa.2011.r10138.
13. Gneiting T, Raftery AE. 2007. Strictly proper scoring rules, prediction, and estimation. J. Amer. Stat. Assoc. 102: 359-378, doi: 10.1198/016214506000001437.
14. Good IJ. 1952. Rational decisions. J. R. Stat. Soc. B 14: 107-114.
15. Hamill TM, Colucci SJ. 1997. Verification of the Eta-RSM short-range ensemble forecasts. Mon. Weather Rev. 125: 1312-1327, doi: 10.1175/1520-0493(1997)125<1312:VOERSR>2.0.CO;2.
16. Mason SJ, Stephenson DB. 2008. How do we know whether seasonal climate forecasts are any good? In Seasonal Climate: Forecasting and Managing Risk, Troccoli A, Harrison M, Anderson DLT, Mason SJ. (eds): 259-289. Springer: Dordrecht, The Netherlands.
17. Mason SJ, Weigel AP. 2009. A generic forecast verification framework for administrative purposes. Mon. Weather Rev. 137: 331-349, doi: 10.1175/2008MWR2553.1.
18. Mason SJ, Galpin JS, Goddard L, Graham NE, Rajartnam B. 2007. Conditional exceedance probabilities. Mon. Weather Rev. 135: 363-372, doi: 10.1175/MWR3284.1.
19. Matheson JE, Winkler RL. 1976. Scoring rules for continuous probability distributions. Manage. Sci. 22: 1087-1096, doi: 10.1287/mnsc.22.10.1087.
20. McCarthy J. 1956. Measures of the value of information. Proc. Natl. Acad. Sci. 42: 654-655.
21. Murphy AH. 1997. Forecast verification. In Economic Value of Weather and Climate Forecasts, Katz RW, Murphy AH. (eds): 19-74. Cambridge University Press: Cambridge, UK.
22. Murphy AH, Daan H. 1985. Forecast evaluation. In Probability, Statistics, and Decision Making in the Atmospheric Sciences, Murphy AH, Katz RW. (eds): 379-437. Westview Press: Boulder, CO.
23. Savage LJ. 1971. Elicitation of personal probabilities and expectations. J. Amer. Stat. Assoc. 66: 783-801, doi: 10.1080/01621459.1971.10482346.
24. Stephenson DB. 2000. Use of the 'odds ratio' for diagnosing forecast skill. Weather and Forecasting 15: 221-232, doi: 10.1175/1520-0434(2000)015<0221:UOTORF>2.0.CO;2.
25. Stephenson DB, Doblas-Reyes FJ. 2000. Statistical methods for interpreting Monte Carlo ensemble forecasts. Tellus 52A: 300-322, doi: 10.1034/j.1600-0870.2000.d01-5.x.
26. Weigel AP. 2012. Verification of ensemble forecasts. In Forecast Verification: a Practitioner's Guide in Atmospheric Science, Jolliffe IT, Stephenson DB. (eds): 141-166. John Wiley and Sons: Chichester, UK.
27. Wilks DS. 2006. Statistical Methods in the Atmospheric Sciences (2nd edn). Academic Press: Amsterdam.
28. Winkler RL. 1967. The quantification of judgment: Some methodological suggestions. J. Amer. Stat. Assoc. 62: 1105-1120, doi: 10.1080/01621459.1967.10500920.
29. Winkler RL, Murphy AH. 1968. 'Good' probability assessors. J. Appl. Meteorol. 7: 751-758, doi: 10.1175/1520-0450(1968)007<0751:PA>2.0.CO;2.