Sampling distributions of the Brier score and Brier skill score under serial dependence

Quarterly Journal of the Royal Meteorological Society

Volumn 136, Issue 653, 2010, Pages 2109-2118

  Wilks, D S ^a  

^a Institute for the Study of the Continents   (United States)

Author keywords

Autocorrelation; Effective sample size; Forecast verification; Probability forecasts

Indexed keywords

AUTOCORRELATED; CONFIDENCE INTERVAL; EVENT PROBABILITY; FORECAST ACCURACY; FORECAST VERIFICATION; HYPOTHESIS TESTS; LOW QUALITIES; P-VALUES; PROBABILITY FORECASTS; SAMPLE SIZES; SAMPLING DISTRIBUTION; SAMPLING PROPERTIES; SERIAL CORRELATION; SERIAL DEPENDENCE; SKILL SCORE;

AUTOCORRELATION; BLIND SOURCE SEPARATION; FORECASTING; STATISTICAL TESTS;

SAMPLING;

AUTOCORRELATION; CLIMATOLOGY; PROBABILITY; SAMPLING; WEATHER FORECASTING;

EID: 78650041423     PISSN: 00359009     EISSN: 1477870X     Source Type: Journal    
DOI: 10.1002/qj.709     Document Type: Article

References (40)

1. Albers W. 1978. Testing the mean of a normal population under dependence. Ann. Stat. 6: 1337-1344.
2. Bradley AA, Schwartz SS, Hashino T. 2008. Sampling uncertainty and confidence intervals for the Brier score and Brier skill score. Weather and Forecasting 23: 992-1006.
3. Bröcker J. 2008. Some remarks on the reliability of categorical probability forecasts. Mon. Weather Rev. 136: 4488-4502.
4. Bröcker J, Smith LA. 2007. Increasing the reliability of reliability diagrams. Weather and Forecasting 22: 651-661.
5. Efron B, Tibshirani RJ. 1993. An introduction to the bootstrap. Monographs on statistics and applied probability 57. Chapman and Hall.
6. Elmore KL. 2005. Alternatives to the chi-square test for evaluating rank histograms from ensemble forecasts. Weather and Forecasting 20: 789-795.
7. Faes C, Molenberghs G, Aerts M, Verbeke G, Kenward MG. 2009. The effective sample size and an alternative small-sample degrees-of-freedom method. Am. Stat. 63: 389-399.
8. Ferro CAT. 2007. Comparing probabilistic forecasting systems with the Brier score. Weather and Forecasting 22: 1076-1088.
9. Hamill TM. 1999. Hypothesis tests for evaluating numerical precipitation forecasts. Weather and Forecasting 14: 155-167.
10. Hanssen AW, Kuipers WJA. 1965. On the relationship between the frequency of rain and various meteorological parameters. Mededeelingen en Verhandelingen 81: 2-15.
11. Hogan RJ, O'Connor EJ, Illingworth AJ. 2009. Verification of cloud-fraction forecasts. Q. J. R. Meteorol. Soc. 135: 1494-1511.
12. Hsu W-R, Murphy AH. 1986. The attributes diagram: A geometrical framework for assessing the quality of probability forecasts. Int. J. Forecasting 2: 285-293.
13. Jolliffe IT. 2007. Uncertainty and inference for verification measures. Weather and Forecasting 22: 637-650.
14. Jolliffe IT, Primo C. 2008. Evaluating rank histograms using decompositions of the chi-square test statistic. Mon. Weather Rev. 136: 2133-2139.
15. Jones RH. 1975. Estimating the variance of time averages. J. Appl. Meteorol. 14: 159-163.
16. Kane TL, Brown BG. 2000. 'Confidence intervals for some verification measures-a survey of several methods.' Pp 46-49 in Preprints, 15th Conference on probability and statistics in the atmospheric sciences. American Meteorological Society.
17. Katz RW. 1982. Statistical evaluation of climate experiments with general circulation models: A parametric time series modeling approach. J. Atmos. Sci. 39: 1446-1455.
18. Leith CE. 1973. The standard error of time-average estimates of climatic means. J. Appl. Meteorol. 12: 1066-1069.
19. Livezey RE. 1995. Field intercomparison. Pp 159-175 in Analysis of climate variability, von Storch H, Navarra A (eds). Springer.
20. Marzban C, Sandgathe S. 2008. Cluster analysis for object-oriented verification of fields: A variation. Mon. Weather Rev. 136: 1013-1025.
21. Mason SJ. 2004. On using 'climatology' as a reference strategy in the Brier and ranked probability skill scores. Mon. Weather Rev. 132: 1891-1895.
22. Mason SJ. 2008. Understanding forecast verification statistics. Meteorol. Appl. 15: 31-40.
23. Mason SJ, Graham NE. 2002. Areas beneath the relative operating characteristic (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. Q. J. R. Meteorol. Soc. 128: 2145-2166.
24. Murphy AH. 1973. A new vector partition of the probability score. J. Appl. Meteorol. 12: 595-600.
25. Murphy AH, Wilks DS. 1998. A case study of the use of statistical models in forecast verification: Precipitation probability forecasts. Weather and Forecasting 13: 795-810.
26. Murphy AH, Winkler RL. 1987. A general framework for forecast verification. Mon. Weather Rev. 115: 1330-1338.
27. Pinson P, McSharry P, Madsen H. 2010. Reliability diagrams for non-parametric density forecasts of continuous variables: Accounting for serial correlation. Q. J. R. Meteorol. Soc. 136: 77-90.
28. Ruth DP, Glahn B, Dagostaro V, Gilbert K. 2009. The performance of MOS in the digital age. Weather and Forecasting 24: 504-519.
29. Seaman RS. 1992. Serial correlation considerations when assessing differences in prediction skill. Aust. Meteorol. Mag. 40: 227-237.
30. Seaman R, Mason I, Woodcock F. 1996. Confidence intervals for some performance measures of yes-no forecasts. Aust. Meteorol. Mag. 45: 49-53.
31. Stephenson DB. 2000. Use of the 'odds ratio' for diagnosing forecast skill. Weather and Forecasting 15: 221-232.
32. Thiébaux HJ, Zwiers FW. 1984. The interpretation and estimation of effective sample size. J. Clim. Appl. Meteorol. 23: 800-811.
33. Thornes JE, Stephenson DB. 2001. How to judge the quality and value of weather forecast products. Meteorol. Appl. 8: 307-314.
34. Trenberth KE. 1984. Some effects of finite sample size and persistence on meteorological statistics. Part I: Autocorrelations. Mon. Weather Rev. 112: 2359-2368.
35. Wilks DS. 1991. Representing serial correlation of meteorological events and forecasts in dynamic decision-analytic models. Mon. Weather Rev. 119: 1640-1662.
36. Wilks DS. 1997. Resampling hypothesis tests for autocorrelated fields. J. Climate 10: 65-82.
37. Wilks DS. 2004. The minimum spanning tree histogram as a verification tool for multidimensional ensemble forecasts. Mon. Weather Rev. 132: 1329-1340.
38. Wilks DS. 2006. Statistical methods in the atmospheric sciences, Second edition. International Geophysics Series 91. Academic Press.
39. Woodcock F. 1976. The evaluation of yes/no forecasts for scientific and administrative purposes. Mon. Weather Rev. 104: 1209-1214.
40. Zwiers FW, Thiébaux HJ. 1987. Statistical considerations for climate experiments. Part I: Scalar tests. J. Clim. Appl. Meteorol. 26: 464-476.