A Score Regression Approach to Assess Calibration of Continuous Probabilistic Predictions

Biometrics

Volumn 66, Issue 4, 2010, Pages 1295-1305

  Held, L ^a   Rufibach, K ^a   Balabdaoui, F ^b  

^a UNIVERSITY OF ZURICH   (Switzerland)

^b Centre National de la Recherche Scientifique   (France)

Author keywords

Calibration; Continuous predictions; Crossvalidation; Regression; Scoring rules

Indexed keywords

CALIBRATION; INTEGRAL EQUATIONS; PROBABILITY DISTRIBUTIONS; REGRESSION ANALYSIS;

CONTINUOUS PREDICTION; CROSS VALIDATION; FORECAST DISTRIBUTION; INTEGRAL TRANSFORM; PREDICTIVE DISTRIBUTIONS; PROBABILISTIC PREDICTION; REGRESSION; SCORING RULES; SIGNIFICANCE TEST; STATISTICAL CONSISTENCIES;

FORECASTING;

CALIBRATION; HISTOGRAM; NUMERICAL MODEL; PROBABILITY; REGRESSION ANALYSIS;

EID: 77955583150     PISSN: 0006341X     EISSN: 15410420     Source Type: Journal    
DOI: 10.1111/j.1541-0420.2010.01406.x     Document Type: Article

References (36)

1. Azzalini, A. and Capitano, A. (1999). Statistical applications of the multivariate skew normal distribution. Journal of the Royal Statistical Society, Series B 61, 579-602.
2. Box, G. (1980). Sampling and Bayes' inference in scientific modelling and robustness. Journal of the Royal Statistical Society, Series A: Statistics in Society 143, 383-425.
3. Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review 78, 1-3.
4. Butler, R. W. (1986). Predictive likelihood inference with applications (with discussion). Journal of the Royal Statistical Society, Series B 48, 1-38.
5. Copas, J. B. (1983). Plotting p against x. Applied Statistics 32, 25-31.
6. Cox, D. R. (1958). Two further applications of a model for binary regression. Biometrika 45, 562-565.
7. Czado, C., Gneiting, T., and Held, L. (2009). Predictive model assessment for count data. Biometrics 65, 1254-1261.
8. Dawid, A. P. (1982). The well-calibrated Bayesian. Journal of the American Statistical Association 77, 605-610.
9. Dawid, A. P. (1984). Present position and potential developments: Some personal views. Statistical theory. The prequential approach. Journal of the Royal Statistical Society, Series A 174, 278-292.
10. Dawid, A. P. and Sebastiani, P. (1999). Coherent dispersion criteria for optimal experimental design. Annals of Statistics 27, 65-81.
11. Fuller, W. A. (1987). Measurement Error Models. New York John Wiley & Sons.
12. Gerds, T. A., Cai, T., and Schumacher, M. (2008). The performance of risk prediction models. Biometrical Journal 50, 457-479.
13. Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102, 359-378.
14. Gneiting, T., Larson, K., Westrick, K., Genton, M. G., and Aldrich, E. (2006). Calibrated probabilistic forecasting at the Stateline wind energy center: The regime-switching space-time method. Journal of the American Statistical Association 101, 968-979.
15. Gneiting, T., Balabdaoui, F., and Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society, Series B 69, 243-268.
16. Gneiting, T., Stanberry, L., Grimit, E., Held, L., and Johnson, N. (2008). Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds. Test 17, 211-235.
17. Graf, E., Schmoor, C., Sauerbrei, W., and Schumacher, M. (1999). Assessment and comparison of prognostic classification schemes for survival data. Statistics in Medicine 18, 2529-2545.
18. Hamill, T. (2001). Interpretation of rank histograms for veryfying ensemble forecasts. Monthly Weather Review 129, 550-560.
19. Harrison, D. A., Brady, A. R., Parry, G. J., Carpenter, J. R., and Rowan, K. (2006). Recalibration of risk prediction models in a large multicenter cohort of admissions to adult, general critical care units in the United Kingdom. Critical Care Medicine 34, 1378-1388.
20. Harrison, D. A., Parry, G., Carpenter, J., Short, A., and Rowan, K. (2007). A new risk prediction model for critical care: The Intensive Care National Audit and Research Centre (ICNARC) model. Critical Care Medicine 35, 1091-1098.
21. Hosmer, D. W. and Lemeshow, S. (1989). Applied Logistic Regression. New York John Wiley & Sons.
22. Labianca, D. A. and Simpson, G. (1996). Statistical analysis of blood- to breath-alcohol ratio data in the logarithm-transformed and non-transformed modes. European Journal of Clinical Chemistry and Clinical Biochemistry 34, 111-117.
23. Leone, F., Nelson, L. S., and Nottingham, R. B. (1961). The folded normal distribution. Technometrics 3, 543-550. Correction in 22, 452.
24. Lix, L. M., Yogendran, M. S., Leslie, W. D., Shaw, S. Y., Baumgartner, R., Bowman, C., Metge, C., Gumel, A., Hux, J., and James, R. C. (2008). Using multiple data features improved the validity of osteoporosis case ascertainment from administrative databases. Journal of Clinical Epidemiology 61, 1250-1260.
25. Mason, S. J., Galpin, J. S., Goddard, L., Graham, N. E., and Rajartnam, B. (2007). Conditional exceedance probabilities. Monthly Weather Review 135, 363-372.
26. Miller, M. E., Hui, S. L., and Tierney, W. M. (1991). Validation techniques for logistic regression models. Statistics in Medicine 10, 1213-1226.
27. Riebler, A. and Held, L. (2009). The analysis of heterogeneous time trends in multivariate age-period-cohort models. Biostatistics
28. Sanders, F. (1963). On subjective probability forecasting. Journal of Applied Meteorology 2, 191-201.
29. Schmid, C. H. and Griffith, J. L. (2005). Multivariate classification rules: Calibration and discrimination. In Encyclopedia of Biostatistics, 2nd edition, P. Armitage and T. Colton (eds), Volume 5, 3491-3497. Chichester, U.K. John Wiley.
30. Schumacher, M., Graf, E., and Gerds, T. (2003). How to assess prognostic models for survival data: A case study in oncology. Methods of Information in Medicine 42, 564-571.
31. Smith, J. Q. (1985). Diagnostic checks of non-standard time series models. Journal of Forecasting 4, 283-291.
32. Spiegelhalter, D. (1986). Probabilistic prediction in patient management and clinical trials. Statistics in Medicine 5, 421-433.
33. StataCorp. (2003). STATA, Reference A-F. Stata Corporation, 8.0 edition. College Station, Texas Stata Press.
34. Steyerberg, E. W. (2009). Clinical Prediction Models. New York Springer.
35. Steyerberg, E. W., Borsboom, G. J., Van Houwelingen, H. C., Eijkemans, M. J., and Habbema, J. D. (2004). Validation and updating of predictive logistic regression models: A study on sample size and shrinkage. Statistics in Medicine 23, 2567-2586.
36. Weisberg, S. (1985). Applied Linear Regression, 2nd edition. New York John Wiley & Sons.