Making and evaluating point forecasts

Journal of the American Statistical Association

Volumn 106, Issue 494, 2011, Pages 746-762

  Gneiting, Tilmann ^a  

^a UNIVERSITY OF HEIDELBERG   (Germany)

Author keywords

Bayes rule; Bregman function; Conditional value at risk (CVaR); Decision theory; Elicitability; Expectile; Mean; Median; Mode; Proper scoring rule; Quantile; Statistical functional

Indexed keywords

EID: 79960137323     PISSN: 01621459     EISSN: None     Source Type: Journal    
DOI: 10.1198/jasa.2011.r10138     Document Type: Article

References (119)

1. Abdous, B., and Theodorescu, R. (1992), "Note on the Spatial Quantile of a Random Vector,"Statistics & Probability Letters, 13, 333-336. [757]
2. Acerbi, C. (2002), "Spectral Measures of Risk: A Coherent Representation of Subjective Risk Aversion,"Journal of Banking and Finance, 26, 1505-1518. [756]
3. Ahlburg, D. A., Chatfield, C., Taylor, S. J., Thompson, P. H., Murphy, A. H., Winkler, R. L., Collopy, F., Armstrong, J. S., and Fildes, R. (1992), "A Commentary on Error Measures,"International Journal of Forecasting, 8, 99-111. [758]
4. Armstrong, J. S. (2001), "Evaluating Forecasting Methods,"in Principles of Forecasting, ed. J. S. Armstrong, Norwell MA: Kluwer, pp. 443-471. [747,757,758]
5. Armstrong, J. S., and Collopy, F. (1992), "Error Measures for Generalizing About Forecasting Methods: Empirical Comparisons,"International Journal of Forecasting, 8, 69-80. [758]
6. Artzner, P., Delbaen, F., Eber, J.-M., and Heath, D. (1999), "Coherent Measures of Risk,"Mathematical Finance, 9, 203-228. [756]
7. Athanasopoulos, G., Hyndman, R. J., Song, H., and Wu, D. C. (2011), "The Tourism Forecasting Competition,"International Journal of Forecasting, 27, 822-844. [758]
8. Banerjee, A., Guo, X., andWang, H. (2005), "On the Optimality of Conditional Expectation as a Bregman Predictor,"IEEE Transactions on Information Theory, 51, 2664-2669. [753,756,757]
9. Bao, Y., Lee, T.-H., and Saltoǧlu, B. (2006), "Evaluating Predictive Performance of Value-at-Risk Models in Emerging Markets: A Reality Check,"Journal of Forecasting, 25, 101-128. [756]
10. Berkowitz, J., and O'Brien, J. (2002), "How Accurate Are Value-at-Risk Models at Commercial Banks?"Journal of Finance, 57, 1093-1111. [756]
11. Bernardo, J. M., and Smith, A. F. M. (2000), Bayesian Theory, Chichester: Wiley. [756]
12. Bollerslev, T. (1986), "Generalized Autoregressive Conditional Heteroscedasticity,"Journal of Econometrics, 31, 307-327. [747]
13. Buja, A., Stuetzle, W., and Shen, Y. (2005), "Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications,"working paper, University of Pennsylvania, available at http://www-stat.wharton.upenn.edu/~buja/PAPERS/paper-proper-scoring.pdf. [754]
14. Caporin, M. (2008), "Evaluating Value-at-Risk Measures in the Presence of Long Memory Conditional Volatility,"Journal of Risk, 10, 79-110. [755]
15. Carbone, R., and Armstrong, J. S. (1982), "Evaluation of Extrapolative Forecasting Methods: Results of a Survey of Academicians and Practitioners,"Journal of Forecasting, 1, 215-217. [746,748]
16. Cervera, J. L., and Muñoz, J. (1996), "Proper Scoring Rules for Fractiles,"in Bayesian Statistics, Vol. 5, eds. J. M. Bernardo, J. O. Berger, A. P. Dawid, and A. F. M. Smith, Oxford: Oxford University Press, pp. 513-519. [751]
17. Chaudhuri, P. (1996), "On a Geometric Notion of Quantiles for Multivariate Data,"Journal of the American Statistical Association, 91, 862-872. [757]
18. Chen, K., Guo, S., Lin, Y., and Ying, Z. (2010), "Least Absolute Relative Error Estimation,"Journal of the American Statistical Association, 105, 1104-1112. [758]
19. Christoffersen, P. F. (2003), Elements of Financial Risk Management, San Diego: Academic Press. [756]
20. Christoffersen, P. F., and Diebold, F. X. (1996), "Further Results on Forecasting and Model Selection Under Asymmetric Loss,"Journal of Applied Econometrics, 11, 561-571. [747]
21. Dawid, A. P. (1984), "Statistical Theory: The Prequential Approach,"Journal of the Royal Statistical Society, Ser. A, 147, 278-292. [746,757]
22. Dawid, A. P. (2007), "The Geometry of Proper Scoring Rules,"Annals of the Institute of Statistical Mathematics, 59, 77-93. [751]
23. Dawid, A. P., and Lauritzen, S. L. (2006), "The Geometry of Decision Theory,"in Proceedings of the Second International Symposium on Information Geometry and Its Applications, University of Tokyo, December 12-16, 2005, pp. 22-28. [758]
24. Dawid, A. P., and Sebastiani, P. (1999), "Coherent Dispersion Criteria for Optimal Experimental Design,"The Annals of Statistics, 27, 65-81. [757]
25. DeGroot, M. H., and Fienberg, S. E. (1983), "The Comparison and Evaluation of Probability Forecasters,"The Statistician: Journal of the Institute of Statisticians, 12, 12-22. [754]
26. Diebold, F. X., and Mariano, R. S. (1995), "Comparing Predictive Accuracy,"Journal of Business & Economic Statistics, 13, 253-263. [757]
27. Diebold, F. X., Gunther, T. A., and Tay, A. S. (1998), "Evaluating Density ForecastsWith Applications to Financial Risk Management,"International Economic Review, 39, 863-883. [757]
28. Duffie, D., and Pan, J. (1997), "An Overview of Value at Risk,"Journal of Derivatives, 4, 7-49. [754]
29. Efron, B. (1991), "Regression Percentiles Using Asymmetric Squared Error Loss,"Statistica Sinica, 1, 93-125. [750]
30. Engelberg, J., Manski, C. F., andWilliams, J. (2009), "Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters,"Journal of Business & Economic Statistics, 27, 30-41. [747,748,759]
31. Engle, R. F. (1982), "Autoregressive Conditional Heteroscedasticity With Estimates of the Variance of United Kingdom Inflation,"Econometrica, 45, 987-1007. [747]
32. Ferguson, T. S. (1967), Mathematical Statistics: A Decision-Theoretic Approach, New York: Academic. [750,754,756]
33. Fildes, R. (1992), "The Evaluation of Extrapolative Forecasting Methods,"International Journal of Forecasting, 8, 81-98. [758]
34. Fildes, R., and Goodwin, P. (2007), "Against Your Better Judgement? How Organizations Can Improve Their Use of Management Judgement in Forecasting,"Interfaces, 37, 570-576. [746,748,757]
35. Fildes, R., Nikolopoulos, K., Crone, S. F., and Syntetos, A. A. (2008), "Forecasting and Operational Research: A Review,"Journal of the Operational Research Society, 59, 1150-1172. [747]
36. French, S., and Ríos Insua, D. (2000), Statistical Decision Theory, London: Arnold. [752,756]
37. Geweke, J., and Whiteman, C. (2006), "Bayesian Forecasting,"in Handbook of Economic Forecasting, Vol. 1, eds. G. Elliott, C.W. J. Granger, and A. Timmermann, Amsterdam: Elsevier, pp. 3-80. [757]
38. Giacomini, R., and Komunjer, I. (2005), "Evaluation and Combination of Conditional Quantile Forecasts,"Journal of Business & Economic Statistics, 23, 416-431. [756]
39. Gneiting, T. (2008a), "Editorial: Probabilistic Forecasting,"Journal of the Royal Statistical Society, Ser. A, 171, 319-321. [746,757]
40. Gneiting, T. (2008b), "Quantiles as Optimal Point Forecasts,"Technical Report 538, University of Washington, Dept. Statistics, available at http://www.stat.washington.edu/research/reports/2008/tr538.pdf. [755]
41. Gneiting, T. (2011), "Quantiles as Optimal Point Forecasts,"International Journal of Forecasting, 27, 197-207. [754]
42. Gneiting, T., and Raftery, A. E. (2007), "Strictly Proper Scoring Rules, Prediction, and Estimation,"Journal of the American Statistical Association, 102, 359-378. [751,754]
43. Gneiting, T., and Thorarinsdottir, T. L. (2010), "Predicting Inflation: Professional Experts versus No-Change Forecasts,"unpublished manuscript, University of Heidelberg, available at http://arxiv.org/abs/1010.2318. [758]
44. Gneiting, T., Stanberry, L. I., Grimit, E. P., Held, L., and Johnson, N. A. (2008), "Assessing Probabilistic Forecasts of Multivariate Quantities, With Applications to Ensemble Predictions of Surface Winds,"Test, 17, 211-264. [756]
45. Granger, C. W. J. (1969), "Prediction With a Generalized Cost of Error Function,"Operational Research Quarterly, 20, 199-207. [756]
46. Granger, C. W. J., and Machina, M. J. (2006), "Forecasting and Decision Theory,"in Handbook of Economic Forecasting, Vol. 1, eds. G. Elliott,C. W. J. Granger, and A. Timmermann, Amsterdam: Elsevier, pp. 82-98. [750,753]
47. Granger, C. W. J., and Pesaran, M. H. (2000a), "Economic and Statistical Measures of Forecast Accuracy,"Journal of Forecasting, 19, 537-560. [750,757]
48. Granger, C. W. J., and Pesaran, M. H. (2000b), "A Decision Theoretic Approach to Forecast Evaluation,"in Statistics and Finance: An Interface, eds. W.-S. Chan, W. K. Li, and H. Tong, London: Imperial College Press, pp. 261-278. [750,757]
49. Hallin, M., Paindaveine, D., and Šiman, M. (2010), "Regression Quantiles: From L1 Optimization to Halfspace Depth"(with discussion), The Annals of Statistics, 38, 635-703. [757]
50. Henderson, R., Jones, M., and Stare, J. (2001), "Accuracy of Point Predictions in Survival Analysis,"Statistics in Medicine, 20, 3083-3096. [756]
51. Hering, A. S., and Genton, M. G. (2010), "Powering UpWith Space-TimeWind Forecasting,"Journal of the American Statistical Association, 105, 92-104. [756]
52. Hilden, J. (2008), "Scoring Rules for Evaluation of Prognosticians and Prognostic Rules,"workshop notes, University of Copenhagen, available at http://biostat.ku.dk/~jh/. [751]
53. Hoerl, A. E., and Kennard, R.W. (1970), "Ridge Regression: Biased Estimation for Nonorthogonal Problems,"Technometrics, 12, 55-67. [758]
54. Horowitz, J. L., and Manski, C. F. (2006): "Identification and Estimation of Statistical Functionals Using Incomplete Data,"Journal of Econometrics, 132, 445-459. [750]
55. Huber, P. J. (1964), "Robust Estimation of a Location Parameter,"Annals of Mathematical Statistics, 35, 73-101. [758]
56. Huber, P. J., and Ronchetti, P. M. (2009), Robust Statistics (2nd ed.), Hoboken, NJ: Wiley. [750,758]
57. Hyndman, R. J., and Koehler, A. B. (2006), "Another Look at Measures of Forecast Accuracy,"International Journal of Forecasting, 22, 679-688. [758]
58. James, W., and Stein, C. (1961), "Estimation With Quadratic Loss,"in Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1, ed. J. Neyman, Berkeley: University of California Press, pp. 361-379. [757]
59. Jolliffe, I. T. (2008), "The Impenetrable Hedge: A Note on Propriety, Equitability and Consistency,"Meteorological Applications, 15, 25-29. [751]
60. Jolliffe, I. T., and Stephenson, D. B. (eds.) (2003), Forecast Verification: A Practitioner's Guide in Atmospheric Science, Chichester: Wiley. [757,758]
61. Jose, V. R. R., and Winkler, R. L. (2009), "Evaluating Quantile Assessments,"Operations Research, 57, 1287-1297. [751]
62. Kemperman, J. H. B. (1987), "The Median of a Finite Measure on a Banach Space,"in Statistical Data Analysis Based on the L1 Norm and Related Methods, ed. Y. Dodge, Amsterdam: North Holland, pp. 217-230. [757]
63. Keynes, J. M. (1911), "The Principal Averages and the Laws of Error Which Lead to Them,"Journal of the Royal Statistical Society, 74, 322-331. [758]
64. Klein, I., and Grottke, M. (2008), "On J. M. Keynes' 'The Principal Averages and the Laws of Error Which Lead to Them'-Refinement and Generalisation,"discussion paper, University of Erlangen, available at http://www.iwqw.wiso.uni-erlangen.de/forschung/07-2008.pdf. [758]
65. Koenker, R. (2005), Quantile Regression, Cambridge: Cambridge University Press. [755]
66. Koenker, R. (2006), "The Median Is the Message: Toward the Fréchet Mean,"Journal de la Société Française de Statistique, 147, 61-64. [757]
67. Koenker, R., and Bassett, G. (1978), "Regression Quantiles,"Econometrica, 46, 33-50. [754,758]
68. Koltchinskii, V. I. (1997), "M-Estimation, Convexity and Quantiles,"The Annals of Statistics, 25, 435-477. [757]
69. Komunjer, I. (2005), "Quasi Maximum-Likelihood Estimation for Conditional Quantiles,"Journal of Econometrics, 128, 137-164. [758]
70. Lambert, N. S., Pennock, D. M., and Shoham, Y. (2008), "Eliciting Properties of Probability Distributions,"in Proceedings of the 9th ACM Conference on Electronic Commerce, July 8-12, 2008, Chicago, Illinois, extended abstract. [749-752,757]
71. Laurent, S., Rombouts, J. V. K., and Violante, F. (2009), "On Loss Functions and Ranking Forecasting Performances of Multivariate Volatility Models,"discussion paper, Catholic University of Louvain, available at http://www.cirpee.org/fileadmin/documents/Cahiers_2009/CIRPEE09-48.pdf. [756,757]
72. Lehmann, E., and Casella, G. (1998), Theory of Point Estimation (2nd ed.), New York: Springer. [756]
73. Lehmann, E. L. (1951), "A General Concept of Unbiasedness,"Annals of Mathematical Statistics, 22, 587-592. [751]
74. Lopez, J. A. (1999), "Regulatory Evaluation of Value-at-Risk Models,"Journal of Risk, 1, 37-64. [755]
75. Makridakis, S., and Hibon, M. (1979), "Accuracy of Forecasting: An Empirical Investigation"(with discussion), Journal of the Royal Statistical Society, Ser. A, 142, 97-145. [758]
76. Makridakis, S., and Hibon, M. (2000), "The M3-Competition: Results, Conclusions and Implications,"International Journal of Forecasting, 16, 451-476. [758]
77. Makridakis, S., Andersen, A., Carbone, R., Fildes, R., Hibon, M., Lewandowski, R., Newton, J., Parzen, E., and Winkler, R. (1982), "The Accuracy of Extrapolation (Time Series) Methods: Results of a Forecasting Competition,"Journal of Forecasting, 1, 111-153. [758]
78. Makridakis, S., Chatfield, C., Hibon, M., Lawrance, M., Mills, T., Ord, K., and Simmons, L. F. (1993), "TheM2-Competition: A Real-Time Judgementally Based Forecasting Study,"International Journal of Forecasting, 9, 5-22. [758]
79. McCarthy, T. M., Davis, D. F., Golicic, S. L., and Mentzner, J. T. (2006), "The Evolution of Sales Forecasting Management: A 20-Year Longitudinal Study of Forecasting Practice,"Journal of Forecasting, 25, 303-324. [746,748]
80. Mentzner, J. T., and Kahn, K. B. (1995), "Forecasting Technique Familiarity, Satisfaction, Usage, and Application,"Journal of Forecasting, 14, 465-476. [746,748]
81. Moskaitis, J. R., and Hansen, J. A. (2006), "Deterministic Forecasting and Verification: A Busted System?"working paper, Massachusetts Institute of Technology, available at http://wind.mit.edu/~hansen/papers/MoskaitisHansenWAF2006.pdf. [747]
82. Murphy, A. H. (1973), "Hedging and Skill Scores for Probability Forecasts,"Journal of Applied Meteorology, 12, 215-223. [758]
83. Murphy, A. H., and Daan, H. (1985), "Forecast Evaluation,"in Probability, Statistics and Decision Making in the Atmospheric Sciences, eds. A. H. Murphy, and R.W. Katz, Boulder, CO:Westview Press, pp. 379-437. [748,751,759]
84. Murphy, A. H., and Winkler, R. L. (1987), "A General Framework for Forecast Verification,"Monthly Weather Review, 115, 1330-1338. [747]
85. Noorbaloochi, S., and Meeden, G. (1983), "Unbiasedness as the Dual of Being Bayes,"Journal of the American Statistical Association, 78, 619-623. [751]
86. Newey, W. K., and Powell, J. L. (1987), "Asymmetric Least Squares Estimation and Testing,"Econometrica, 55, 819-847. [755]
87. Osband, K. (2011), Pandora's Risk: Uncertainty at the Core of Finance, New York: Columbia University Press. [756]
88. Osband, K., and Reichelstein, S. (1985), "Information-Eliciting Compensation Schemes,"Journal of Public Economics, 27, 107-115. [756,757]
89. Osband, K. H. (1985), "Providing Incentives for Better Cost Forecasting,"Ph.D. thesis, University of California, Berkeley. [749-753,759]
90. Park, H., and Stefanski, L. A. (1998), "Relative-Error Prediction,"Statistics & Probability Letters, 40, 227-236. [752]
91. Park, T., and Casella, G. (2008), "The Bayesian Lasso,"Journal of the American Statistical Association, 103, 681-686. [758]
92. Patton, A. J. (2011), "Volatility Forecast Comparison Using Imperfect Volatility Proxies,"Journal of Econometrics, 160, 246-256. [746,750,752-755,760]
93. Patton, A. J., and Sheppard, K. (2009), "Evaluating Volatility and Correlation Forecasts,"in Handbook of Financial Time Series, eds. T. G. Anderson, R. A. Davis, J.-P. Kreiss, and T. Mikosch, Berlin: Springer, pp. 801-838. [757]
94. Pers, T. H., Albrechtsen, A., Holst, C., Sørensen, T. I. A., and Gerds, T. A. 2009), "The Validation and Assessment of Machine Learning: A Game of Prediction From High-Dimensional Data,"PLoS ONE, 4, e6287, doi:10.1371/journal.pone.0006287. [758]
95. Pesaran, M. H., and Skouras, S. (2002), "Decision-Based Methods for Forecast Evaluation,"in A Companion to Economic Forecasting, eds. M. H. Pesaran and S. Skouras, Malden: Blackwell Publishers, pp. 241-267. [751]
96. Phelps, R. R. (1966), Lectures on Choquet's Theorem, Princeton: D. Van Nostrand. [751]
97. Pratt, J. W., Raiffa, H., and Schlaifer, R. (1995), Introduction to Statistical Decision Theory, Boston: Massachusetts Institute of Technology. [759]
98. Pukelsheim, F. (2006), Optimal Design of Experiments (SIAM Classics ed.), Philadelphia: SIAM. [757]
99. Raiffa, H., and Schlaifer, R. (1961), Applied Statistical Decision Theory, Clinton: Colonial Press. [754]
100. Reichelstein, S., and Osband, K. (1984), "Incentives in Government Contracts,"Journal of Public Economics, 24, 257-270. [753]
101. Rockafellar, R. T. (1970), Convex Analysis, Princeton: Princeton University Press. [753]
102. Rockafellar, R. T., and Uryasev, S. (2000), "Optimization of Conditional Valueat-Risk,"Journal of Risk, 2, 21-42. [756]
103. Rockafellar, R. T., and Uryasev, S. (2002), "Conditional Value-at-Risk for General Loss Distributions,"Journal of Banking and Finance, 26, 1443-1471. [756]
104. Saerens, M. (2000), "Building Cost Functions Minimizing to Some Summary Statistics,"IEEE Transactions on Neural Networks, 11, 1263-1271. [749,753,754]
105. Savage, L. J. (1971), "Elicitation of Personal Probabilities and Expectations,"Journal of the American Statistical Association, 66, 783-810. [749,753,754,756,757,759]
106. Schervish, M. J. (1989), "A General Method for Comparing Probability Assessors,"The Annals of Statistics, 17, 1856-1879. [754]
107. Serfling, R. (2002), "Quantile Functions for Multivariate Analysis: Approaches and Applications,"Statistica Neerlandica, 56, 214-232. [757]
108. Shipp, S., and Cohen, S. (2009), "COPAFS Focuses on Statistical Activities,"Amstat News, August 2009, 15-18. [758]
109. Small, C. G. (1990), "A Survey of Multidimensional Medians,"International Statistical Review, 58, 263-277. [757]
110. Thomson,W. (1979), "Eliciting Production Possibilities From aWell-Informed Manager,"Journal of Economic Theory, 20, 360-380. [749,753,754,757]
111. Wedderburn, R. W. M. (1974), "Quasi-Likelihood Functions, Generalized Linear Models, and the Gauss-Newton Method,"Biometrika, 61, 439-447. [758]
112. Wellner, J. A. (2009), "Statistical Functionals and the Delta Method,"lecture notes, University of Washington, available at http://www.stat.washington.edu/ people/jaw/COURSES/580s/581/LECTNOTES/ch7.pdf. [750]
113. West, K. D. (1996), "Asymptotic Inference About Predictive Ability,"Econometrica, 64, 1067-1084. [757]
114. West, M., and Harrison, J. (1997), Bayesian Forecasting and Dynamic Models (2nd ed.), New York: Springer. [757]
115. Winkler, R. L. (1996), "Scoring Rules and the Evaluation of Probabilities"(with discussion), Test, 5, 1-60. [751,754]
116. Yelland, P. M., Kim, S., and Stratulate, R. (2010), "A Bayesian Model for Sales Forecasting at Sun Microsystems,"Interfaces, 40, 118-129. [758]
117. Yu, C. W. (2009), "Median Loss Analysis and Its Application to Model Selection,"Ph.D. thesis, Dept. of Statistics, University of British Columbia, Vancouver. [758]
118. Zellner, A. (1986), "Bayesian Estimation and Prediction Using Asymmetric Loss Functions,"Journal of the American Statistical Association, 81, 446-451. [758]
119. Zhang, C., Jiang, Y., and Chai, Y. (2010), "Penalized Bregman Divergence for Large-Dimensional Regression and Classification,"Biometrika, 97, 551-566. [758]