Strictly proper scoring rules, prediction, and estimation

Journal of the American Statistical Association

Volumn 102, Issue 477, 2007, Pages 359-378

  Gneiting, Tilmann ^a   Raftery, Adrian E ^a  

^a UNIVERSITY OF WASHINGTON   (United States)

Author keywords

Bayes factor; Bregman divergence; Brier score; Coherent; Continuous ranked probability score; Cross validation; Entropy; Kernel score; Loss function; Minimum contrast estimation; Negative definite function

Indexed keywords

EID: 33947274775     PISSN: 01621459     EISSN: None     Source Type: Journal    
DOI: 10.1198/016214506000001437     Document Type: Article

References (128)

1. Baringhaus, L., and Franz, C. (2004), "On a New Multivariate Two-Sample Test," Journal of Multivariate Analysis, 88, 190-206.
2. Bauer, H. (2001), Measure and Integration Theory, Berlin: Walter de Gruijter.
3. Berg, C., Christensen, J. P. R., and Ressel, P. (1984), Harmonic Analysis on Semigroups, New York: Springer-Verlag.
4. Bernardo, J. M. (1979), "Expected Information as Expected Utility," The Annals of Statistics, 7, 686-690.
5. Bernardo, J. M., and Smith, A. F. M. (1994), Bayesian Theory, New York: Wiley.
6. Besag, J., Green, P., Higdon, D., and Mengersen, K. (1995), "Bayesian Computing and Stochastic Systems," Statistical Science, 10, 3-66.
7. Birgé, L., and Massart, P. (1993), "Rates of Convergence for Minimum Contrast Estimators," Probability Theory and Related Fields, 97, 113-150.
8. Bregman, L. M. (1967), "The Relaxation Method of Finding the Common Point of Convex Sets and Its Application to the Solution of Problems in Convex Programming," USSR Computational Mathematics and Mathematical Physics. 7, 200-217.
9. Bremnes, J. B. (2004), "Probabilistic Forecasts of Precipitation in Terms of Quantiles Using NWP Model Output," Monthly Weather Review, 132, 338-347.
10. Brier, G. W. (1950), "Verification of Forecasts Expressed in Terms of Probability," Monthly Weather Review, 78, 1-3.
11. Briggs, W. (2005), "A General Method of Incorporating Forecast Cost and Loss in Value Scores," Monthly Weather Review, 133, 3393-3397.
12. Briggs, W., and Ruppert, D. (2005), "Assessing the Skill of Yes/No Predictions," Biometrics, 61, 799-807.
13. Buja, A., Logan, B. F., Reeds, J. A., and Shepp, L. A. (1994), "Inequalities and Positive-Definite Functions Arising From a Problem in Multidimensional Scaling," The Annals of Statistics, 22, 406-438.
14. Buja, A., Stuetzle, W., and Shen, Y. (2005), "Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications," manuscript, available at www-stat.wharton.upenn.edu/ ~buja/.
15. Campbell, S. D., and Diebold, F. X. (2005), "Weather Forecasting for Weather Derivatives," Journal of the American Statistical Association, 100, 6-16.
16. Candille, G., and Talagrand, O. (2005), "Evaluation of Probabilistic Prediction Systems for a Scalar Variable," Quarterly Journal of the Royal Meteorological Society, 131, 2131-2150.
17. Casella, G., Hwang, J. T. G., and Robert, C. (1993), "A Paradox in Decision-Theoretic Interval Estimation," Statistica Sinica, 3, 141-155.
18. Cervera, J. L., and Muñoz, J. (1996), "Proper Scoring Rules for Fractiles," in Bayesian Statistics 5, eds. J. M. Bernardo, J. O. Berger, A. P. Dawid, and A. F. M. Smith, Oxford, U.K.: Oxford University Press, pp. 513-519.
19. Christoffersen, P. F. (1998), "Evaluating Interval Forecasts," International Economic Review, 39, 841-862.
20. Collins, M., Schapire, R. E., and Singer, J. (2002), "Logistic Regression, AdaBoost and Bregman Distances," Machine Learning, 48, 253-285.
21. Copas, J. B. (1983), "Regression, Prediction and Shrinkage," Journal of the Royal Statistical Society, Ser. B, 45, 311-354.
22. Daley, D. J., and Vere-Jones, D. (2004), "Scoring Probability Forecasts for Point Processes: The Entropy Score and Information Gain," Journal of Applied Probability, 41A, 297-312.
23. Dawid, A. P. (1984), "Statistical Theory: The Prequential Approach," Journal of the Royal Statistical Society, Ser. A, 147, 278-292.
24. _ (1986), "Probability Forecasting," in Encyclopedia of Statistical Sciences, Vol. 7, eds. S. Kotz, N. L. Johnson, and C. B. Read, New York: Wiley, pp. 210-218.
25. _ (1998), "Coherent Measures of Discrepancy, Uncertainty and Dependence, With Applications to Bayesian Predictive Experimental Design," Research Report 139, University College London, Dept. of Statistical Science.
26. _ (2006), "The Geometry of Proper Scoring Rules," Research Report 268, University College London, Dept. of Statistical Science.
27. Dawid, A. P., and Sebastiani, P. (1999), "Coherent Dispersion Criteria for Optimal Experimental Design," The Annals of Statistics, 27, 65-81.
28. Deque, M., Royer, J. T., and Stroe, R. (1994), "Formulation of Gaussian Probability Forecasts Based on Model Extended-Range Integrations," Tellus, Ser. A, 46, 52-65.
29. Diebold, F. X., and Mariano, R. S. (1995), "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, 13, 253-263.
30. Duffie, D., and Pan, J. (1997), "An Overview of Value at Risk," Journal of Derivatives, 4, 7-49.
31. Dunsmore, I. R. (1968), "A Bayesian Approach to Calibration," Journal of the Royal Statistical Society, Ser. B, 30, 396-405.
32. Eaton, M. L. (1982), "A Method for Evaluating Improper Prior Distributions," in Statistical Decision Theory and Related Topics III, eds. S. S. Gupta and J. O. Berger, New York: Academic Press, pp. 329-352.
33. Eaton, M. L., Giovagnoli, A., and Sebastiani, P. (1996), "A Predictive Approach to the Bayesian Design Problem With Application to Normal Regression Models," Biometrika, 83, 111-125.
34. Epstein, E. S. (1969), "A Scoring System for Probability Forecasts of Ranked Categories," Journal of Applied Meteorology, 8, 985-987.
35. Feuerverger, A., and Rahman, S. (1992), "Some Aspects of Probability Forecasting," Communications in Statistics - Theory and Methods, 21, 1615-1632.
36. Friederichs, P., and Hense, A. (2006), "Statistical Down-Scaling of Extreme Precipitation Events Using Censored Quantile Regression," Monthly Weather Review, in press.
37. Friedman, D. (1983), "Effective Scoring Rules for Probabilistic Forecasts," Management Science, 29, 447-454.
38. Friedman, J. H. (1989), "Regularized Discriminant Analysis," Journal of the American Statistical Association, 84, 165-175.
39. Garratt, A., Lee, K., Pesaran, M. H., and Shin, Y. (2003), "Forecast Uncertainties in Macroeconomic Modelling: An Application to the U.K. Economy," Journal of the American Statistical Association, 98, 829-838.
40. Garthwaite, P. H., Kadane, J. B., and O'Hagan, A. (2005), "Statistical Methods for Eliciting Probability Distributions," Journal of the American Statistical Association, 100, 680-700.
41. Geisser, S., and Eddy, W. F. (1979), "A Predictive Approach to Model Selection," Journal of the American Statistical Association, 74, 153-160.
42. Gelfand, A. E., and Ghosh, S. K. (1998), "Model Choice: A Minimum Posterior Predictive Loss Approach," Biometrika, 85, 1-11.
43. Gerds, T. (2002), "Nonparametric Efficient Estimation of Prediction Error for Incomplete Data Models," unpublished doctoral dissertation, AlbertLudwigs-Universität Freiburg, Germany, Mathematische Fakultät.
44. Giacomini, R., and Komunjer, I. (2005), "Evaluation and Combination of Conditional Quantile Forecasts," Journal of Business & Economic Statistics, 23, 416-431.
45. Gneiting, T. (1998), "Simple Tests for the Validity of Correlation Function Models on the Circle," Statistics & Probability Letters, 39, 119-122.
46. Gneiting, T., Balabdaoui, F., and Raftery, A. E. (2006), "Probabilistic Forecasts, Calibration and Sharpness," Journal of the Royal Statistical Society, Ser. B, in press.
47. Gneiting, T., and Raftery, A. E. (2005), "Weather Forecasting With Ensemble Methods," Science, 310, 248-249.
48. Gneiting, T., Raftery, A. E., Balabdaoui, F., and Westveld, A. (2003), "Verifying Probabilistic Forecasts: Calibration and Sharpness," presented at the Workshop on Ensemble Forecasting, Val-Morin, Québec.
49. Gneiting, T., Raftery, A. E., Westveld, A., and Goldman, T. (2005), "Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation," Monthly Weather Review, 133, 1098-1118.
50. Good, I. J. (1952), "Rational Decisions," Journal of the Royal Statistical Society, Ser. B, 14, 107-114.
51. _ (1971), Comment on "Measuring Information and Uncertainty," by R. J. Buehler, in Foundations of Statistical Inference, eds. V. P. Godambe and D. A. Sprott, Toronto: Holt, Rinehart and Winston, pp. 337-339.
52. Granger, C. W. J. (2006), "Preface: Some Thoughts on the Future of Forecasting," Oxford Bulletin of Economics and Statistics, 67S, 707-711.
53. Grimit, E. P., Gneiting, T., Berrocal, V. J., and Johnson, N. A. (2006), "The Continuous Ranked Probability Score for Circular Variables and Its Application to Mesoscale Forecast Ensemble Verification," Quarterly Journal of the Royal Meteorological Society, in press.
54. Grimit, E. P., and Mass, C. F. (2002), "Initial Results of a Mesoscale Short-Range Ensemble System Over the Pacific Northwest," Weather and Forecasting, 17, 192-205.
55. Grünwald, P. D., and Dawid, A. P. (2004), "Game Theory, Maximum Entropy, Minimum Discrepancy and Robust Bayesian Decision Theory," The Annals of Statistics, 32, 1367-1433.
56. Gschlößl, S., and Czado, C. (2005), "Spatial Modelling of Claim Frequency and Claim Size in Insurance," Discussion Paper 461, Ludwig-Maximilians-Universität, Munich, Germany, Sonderforschungsbereich 368.
57. Hamill, T. M. (1999), "Hypothesis Tests for Evaluating Numerical Precipitation Forecasts," Weather and Forecasting, 14, 155-167.
58. Hamill, T. M., and Wilks, D. S. (1995), "A Probabilistic Forecast Contest and the Difficulty in Assessing Short-Range Forecast Uncertainty," Weather and Forecasting, 10, 620-631.
59. Hendrickson, A. D., and Buehler, R. J. (1971), "Proper Scores for Probability Forecasters," The Annals of Mathematical Statistics, 42, 1916-1921.
60. Hersbach, H. (2000), "Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems," Weather and Forecasting, 15, 559-570.
61. Hofmann, T., Schölkopf, B., and Smola, A. (2005), "A Review of RKHS Methods in Machine Learning," preprint.
62. Huber, P. J. (1964), "Robust Estimation of a Location Parameter," The Annals of Mathematical Statistics, 35, 73-101.
63. _ (1967), "The Behavior of Maximum Likelihood Estimates Under Non-Standard Conditions," in Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability I, eds. L. M. Le Cam and J. Neyman, Berkeley, CA: University of California Press, pp. 221-233.
64. _ (1981), Robust Statistics, New York: Wiley.
65. Jeffreys, H. (1939), Theory of Probability, Oxford, U.K.: Oxford University Press.
66. Jolliffe, I. T. (2006), "Uncertainty and Inference for Verification Measures," Weather and Forecasting, in press.
67. Jolliffe, I. T., and Stephenson, D. B. (eds.) (2003), Forecast Verification: A Practicioner's Guide in Atmospheric Science, Chichester, U.K.: Wiley.
68. Kabaila, P. (1999), "The Relevance Property for Prediction Intervals," Journal of Time Series Analysis, 20, 655-662.
69. Kabaila, P., and He, Z. (2001), "On Prediction Intervals for Conditionally Heteroscedastic Processes," Journal of Time Series Analysis, 22, 725-731.
70. Kass, R. E., and Raftery, A. E. (1995), "Bayes Factors," Journal of the American Statistical Association, 90, 773-795.
71. Knorr-Held, L., and Rainer, E. (2001), "Projections of Lung Cancer in West Germany: A Case Study in Bayesian Prediction," Biostatistics, 2, 109-129.
72. Koenker, R., and Bassett, G. (1978), "Regression Quantiles," Econometrica, 46, 33-50.
73. Koenker, R., and Machado, J. A. F. (1999), "Goodness-of-Fit and Related Inference Processes for Quantile Regression," Journal of the American Statistical Association. 94, 1296-1310.
74. Kohonen, J., and Suomela, J. (2006), "Lessons Learned in the Challenge: Making Predictions and Scoring Them," in Machine Learning Challenges: Evaluating Predictive Uncertainty, Visual Object Classification, and Recognizing Textual Entailment, eds. J. Quiñonero-Candela, I. Dagan, B. Magnini, and F. d'Alché-Buc, Berlin: Springer-Verlag, pp. 95-116.
75. Koldobskiǐ, A. L. (1992), "Schoenberg's Problem on Positive Definite Functions," St. Petersburg Mathematical Journal, 3, 563-570.
76. Krzysztofowicz, R., and Sigrest, A. A. (1999), "Comparative Verification of Guidance and Local Quantitative Precipitation Forecasts: Calibration Analyses," Weather and Forecasting, 14, 443-454.
77. Langland, R. H., Toth, Z., Gelaro, R., Szunyogh, I., Shapiro, M. A., Majumdar, S. J., Morss, R. E., Rohaly, G. D., Velden, C., Bond, N., and Bishop, C. H. (1999), "The North Pacific Experiment (NORPEX-98): Targeted Observations for Improved North American Weather Forecasts," Bulletin of the American Meteorological Society, 90, 1363-1384.
78. Laud, P. W., and Ibrahim, J. G. (1995), "Predictive Model Selection," Journal of the Royal Statistical Society, Ser, B, 57, 247-262.
79. Lehmann, E., and Casella, G. (1998), Theory of Point Estimation (2nd ed.), New York: Springer.
80. Liu, R. Y. (1990), "On a Notion of Data Depth Based on Random Simplices," The Annals of Statistics, 18, 405-414.
81. Ma, C. (2003), "Nonstationary Covariance Functions That Model Space-Time Interactions," Statistics & Probability Letters, 61, 411-419.
82. Mason, S. J. (2004), "On Using Climatology as a Reference Strategy in the Brier and Ranked Probability Skill Scores," Monthly Weather Review, 132, 1891-1895.
83. Matheron, G. (1984), "The Selectivity of the Distributions and the 'Second Principle of Geostatistics,' "in Geostatistics for Natural Resources Characterization, eds. G. Verly, M. David, and A. G. Journel, Dordrecht: Reidel. pp. 421-434.
84. Matheson, J. E., and Winkler, R. L. (1976), "Scoring Rules for Continuous Probability Distributions," Management Science, 22, 1087-1096.
85. Mattner, L. (1997), "Strict Defmiteness via Complete Monotonicity of Integrals," Transactions of the American Mathematical Society, 349, 3321-3342.
86. McCarthy, J. (1956), "Measures of the Value of Information," Proceedings of the National Academy of Sciences, 42, 654-655.
87. Murphy, A. H. (1973), "Hedging and Skill Scores for Probability Forecasts," Journal of Applied Meteorology, 12, 215-223.
88. Murphy, A. H., and Winkler, R. L. (1992), "Diagnostic Verification of Probability Forecasts," International Journal of Forecasting, 7, 435-455.
89. Nau, R. F. (1985), "Should Scoring Rules Be 'Effective'?," Management Science, 31, 527-535.
90. Palmer, T. N. (2002), "The Economic Value of Ensemble Forecasts as a Tool for Risk Assessment: From Days to Decades," Quarterly Journal of the Royal Meteorological Society, 128, 747-774.
91. Pepe, M. S. (2003), The Statistical Evaluation of Medical Tests for Classification and Prediction, Oxford, U.K.: Oxford University Press.
92. Perlman, M. D. (1972), "On the Strong Consistency of Approximate Maximum Likelihood Estimators," in Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability I, eds. L. M. Le Cam, J. Neyman, and E. L. Scott, Berkeley, CA: University of California Press, pp. 263-281.
93. Pfanzagl, J. (1969), "On the Measurability and Consistency of Minimum Contrast Estimates," Metrika, 14, 249-272.
94. Potts, J. (2003), "Basic Concepts," in Forecast Verification: A Practicioner's Guide in Atmospheric Science, eds. I. T. Jolliffe and D. B. Stephenson, Chichester, U.K.: Wiley, pp. 13-36.
95. Quiñonero-Candela, J., Rasmussen, C. E., Sinz, F., Bousquet, O., and Schölkopf, B. (2006), "Evaluating Predictive Uncertainty Challenge," in Machine Learning Challenges: Evaluating Predictive Uncertainty. Visual Object Classification, and Recognizing Textual Entailment, eds. J. Quiñonero-Candela, I. Dagan, B. Magnini, and F. d'Alché-Buc, Berlin: Springer, pp. 1-27.
96. Raftery, A. E., Gneiting, T., Balabdaoui, F., and Polakowski, M. (2005), "Using Bayesian Model Averaging to Calibrate Forecast Ensembles," Monthly Weather Review, 133, 1155-1174.
97. Rockafellar, R. T. (1970), Convex Analysis, Princeton, NJ: Princeton University Press.
98. Roulston, M. S., and Smith, L. A. (2002), "Evaluating Probabilistic Forecasts Using Information Theory," Monthly Weather Review, 130, 1653-1660.
99. Savage, L. J. (1971), "Elicitation of Personal Probabilities and Expectations," Journal of the American Statistical Association, 66, 783-801.
100. Schervish, M. J. (1989), "A General Method for Comparing Probability Assessors," The Annals of Statistics, 17, 1856-1879.
101. Schumacher, M., Graf, E., and Gerds, T. (2003), "How to Assess Prognostic Models for Survival Data: A Case Study in Oncology," Methods of Information in Medicine, 42, 564-571.
102. Schwarz, G. (1978), "Estimating the Dimension of a Model," The Annals of Statistics, 6, 461-464.
103. Selten, R. (1998), "Axiomatic Characterization of the Quadratic Scoring Rule," Experimental Economics, 1, 43-62.
104. Shuford, E. H., Albert, A., and Massengil, H. E. (1966), "Admissible Probability Measurement Procedures," Psychometrika, 31, 125-145.
105. Smyth, P. (2000), "Model Selection for Probabilistic Clustering Using Cross-Validated Likelihood," Statistics and Computing, 10, 63-72.
106. Spiegelhalter, D. J., Best, N. G., Carlin, B. R., and van der Linde, A. (2002), "Bayesian Measures of Model Complexity and Fit" (with discussion and rejoinder), Journal of the Royal Statistical Society, Ser. B, 64, 583-616.
107. Staël von Holstein, C.-A. S. (1970), "A Family of Strictly Proper Scoring Rules Which Are Sensitive to Distance," Journal of Applied Meteorology, 9, 360-364.
108. _ (1977), "The Continuous Ranked Probability Score in Practice," in Decision Making and Change in Human Affairs, eds. H. Jungermann and G. de Zeeuw, Dordrecht: Reidel, pp. 263-273.
109. Székely, G. J. (2003), "ε-Statistics: The Energy of Statistical Samples," Technical Report 2003-16, Bowling Green State University, Dept. of Mathematics and Statistics.
110. Székely, G. J., and Rizzo, M. L. (2005), "A New Test for Multivariate Normality," Journal of Multivariate Analysis, 93, 58-80.
111. Taylor, J. W. (1999), "Evaluating Volatility and Interval Forecasts," Journal of Forecasting, 18, 111-128.
112. Tetlock, P. E. (2005), Political Expert Judgement, Princeton, NJ: Princeton University Press.
113. Theis, S. (2005), "Deriving Probabilistic Short-Range Forecasts From a Deterministic High-Resolution Model," unpublished doctoral dissertation, Rheinische Friedrich-Wilhelms-Universität, Bonn, Germany, MathematischNaturwissenschaftliche Fakultät.
114. Toth, Z., Zhu, Y., and Marchok, T. (2001), "The Use of Ensembles to Identify Forecasts With Small and Large Uncertainty," Weather and Forecasting, 16, 463-477.
115. Unger, D. A. (1985), "A Method to Estimate the Continuous Ranked Probability Score," in Preprints of the Ninth Conference on Probability and Statistics in Atmospheric Sciences, Virginia Beach, Virginia, Boston: American Meteorological Society, pp. 206-213.
116. Wald, A. (1949), "Note on the Consistency of the Maximum Likelihood Estimate," The Annals of Mathematical Statistics, 20, 595-601.
117. Weigend, A. S., and Shi, S. (2000), "Predicting Daily Probability Distributions of S&P500 Returns," Journal of Forecasting, 19, 375-392.
118. Wilks, D. S. (2002), "Smoothing Forecast Ensembles With Fitted Probability Distributions," Quarterly Journal of the Royal Meteorological Society, 128, 2821-2836.
119. _ (2006), Statistical Methods in the Atmospheric Sciences (2nd ed.), Amsterdam: Elsevier.
120. Wilson, L. J., Burrows, W. R., and Lanzinger, A. (1999), "A Strategy for Verification of Weather Element Forecasts From an Ensemble Prediction System," Monthly Weather Review, 127, 956-970.
121. Winkler, R. L. (1969), "Scoring Rules and the Evaluation of Probability Assessors," Journal of the American Statistical Association, 64, 1073-1078.
122. _ (1972), "A Decision-Theoretic Approach to Interval Estimation," Journal of the American Statistical Association, 67, 187-191.
123. _ (1994), "Evaluating Probabilities: Asymmetric Scoring Rules," Management Science, 40, 1395-1405.
124. _ (1996), "Scoring Rules and the Evaluation of Probabilities" (with discussion and reply), Test, 5, 1-60.
125. Winkler, R. L., and Murphy, A. H. (1968), "'Good' Probability Assessors," Journal of Applied Meteorology, 7, 751-758.
126. _ (1979), "The Use of Probabilities in Forecasts of Maximum and Minimum Temperatures," Meteorological Magazine, 108, 317-329.
127. Zastavnyi, V. P. (1993), "Positive Definite Functions Depending on the Norm," Russian Journal of Mathematical Physics, 1, 511-522.
128. Zuo, Y., and Serfling, R. (2000), "General Notions of Statistical Depth Functions," The Annals of Statistics, 28, 461-482.