Assessing calibration of multinomial risk prediction models

Statistics in Medicine

Volumn 33, Issue 15, 2014, Pages 2585-2596

  Van Hoorde, Kirsten ^a   Vergouwe, Yvonne ^b   Timmerman, Dirk ^a,c   Van Huffel, Sabine ^a   Steyerberg, Ewout W ^b   Van Calster, Ben ^a  

^a UNIVERSITY OF LEUVEN   (Belgium)

^b ERASMUS MEDICAL CENTER   (Netherlands)

^c UNIVERSITY HOSPITALS LEUVEN   (Belgium)

Author keywords

Calibration; Calibration plot; Polytomous diagnosis; Risk prediction model

Indexed keywords

CISPLATIN;

ARTICLE; CALCULATION; CALIBRATION; CANCER CHEMOTHERAPY; CANCER DIAGNOSIS; CONTROLLED STUDY; HUMAN; LINEAR SYSTEM; LOGISTIC REGRESSION ANALYSIS; MAJOR CLINICAL STUDY; MATHEMATICAL COMPUTING; OVARY TUMOR; PREDICTION; RISK ASSESSMENT; STATISTICAL MODEL; TESTIS CANCER; ADULT; FEMALE; MALE; MIDDLE AGED; PATHOLOGY; PROCEDURES; STANDARDS; TESTICULAR NEOPLASMS;

ADULT; CISPLATIN; FEMALE; HUMANS; LOGISTIC MODELS; MALE; MIDDLE AGED; OVARIAN NEOPLASMS; RISK ASSESSMENT; TESTICULAR NEOPLASMS;

EID: 84901941477     PISSN: 02776715     EISSN: 10970258     Source Type: Journal    
DOI: 10.1002/sim.6114     Document Type: Article

References (43)

1. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21(1):128-138. DOI: 10.1097/EDE.0b013e3181c30fb2.
2. Viallon V, Ragusa S, Clavel-Chapelon F, Bénichou J. How to evaluate the calibration of a disease risk prediction tool. Statistics in Medicine 2009; 28:901-916. DOI: 10.1002/sim.3517.
3. Harrell F. Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer: UK, 2001.
4. Peek N, Arts D, Bosman R, van der Voort P, de Keizer N. External validation of prognostic models for critically ill patients required substantial sample sizes. Journal of Clinical Epidemiology 2007; 60:491-501. DOI: 10.1016/j.jclinepi.2006.08.011.
5. Vergouwe Y, Steyerberg E, Eijkemans M, Habbema J. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. Journal of Clinical Epidemiology 2005; 58(5):475-483. DOI: 10.1016/j.jclinepi.2004.06.017.
6. Cox D. Two further applications of a model for binary regression. Biometrika 1958; 45:562-565. DOI: 10.2307/2333203.
7. Miller ME, Langefeld CD, Tierney WM, Hui SL, McDonald CJ. Validation of probabilistic predictions. Medical Decision Making 1993; 13:49-58. DOI: 10.1177/0272989X9301300107.
8. Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer: United States of America, 2009.
9. Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Statistics in Medicine 2014; 33:517-535. DOI: 101002/sim.5941.
10. Dalton J. Flexible recalibration of binary clinical prediction models. Statistics in Medicine 2013; 32(2):282-289. DOI: 10.1002/sim.5544.
11. Mossman D. Three-way ROCs. Medical Decision Making 1999; 19:78-89. DOI: 10.1177/0272989X9901900110.
12. Nakas C, Alonzo T. ROC graphs for assessing the ability of a diagnostic marker to detect three disease classes with an umbrella ordering. Biometrics 2007; 63:603-609. DOI: 10.1111/j.1541-0420.2006.00715.x.
13. Obuchowski NA, Goske MJ, Applegate KE. Assessing physicians' accuracy in diagnosing paediatric patients with acute abdominal pain: measuring accuracy for multiple diseases. Statistics in Medicine 2001; 20:3261-3278. DOI: 10.1002/sim.944.
14. Van Calster B, Van Belle V, Vergouwe Y, Steyerberg E. Discrimination ability of prediction models for ordinal outcomes: relationships between existing measures and a new measure. Biometrical Journal 2012; 54(5):674-685. DOI: 10.1002/bimj.201200026.
15. Van Calster B, Van Belle V, Vergouwe Y, Timmerman D, Van Huffel S, Steyerberg E. Extending the c statistic to nominal polytomous outcomes: the polytomous discrimination index. Statistics in Medicine 2012; 31:2610-2626. DOI: 10.1002/sim.5321.
16. Van Calster B, Vergouwe Y, Van Belle V, Looman C, Timmerman D, Steyerberg E. Assessing the discriminative ability of risk models for more than two outcome categories: a perspective. European Journal of Epidemiology 2012; 27(10):761-770. DOI: 10.1007/s10654-012-9733-3.
17. Fagerland M, Hosmer D, Bofin A. Multinomial goodness-of-fit tests for logistic regression models. Statistics in Medicine 2008; 27(21):4238-4253. DOI: 10.1002/sim.3202.
18. Goeman J, le Cessie S. A goodness-of-fit test for multinomial logistic regression. Biometrics 2006; 62(4):980-985. DOI: 10.1111/j.1541-0420.2006.00581.x.
19. Pigeon J, Heyse J. An improved goodness of fit statistic for probability prediction models. Biometrical Journal 1999; 41:71-82. DOI: 10.1002/(SICI)1521-4036(199903)41:1<71::AID-BIMJ71>3.0.CO;2-O.
20. Hosmer D, Hosmer T, Le Cessie S, Lemeshow S. A comparison of goodness-of-fit tests for the logistic regression models. Statistics in Medicine 1997; 16:965-980. DOI: 10.1002/(SICI)1097-0258(19970515)16:9<965::AID-SIM509>3.0.CO;2-O.
21. Hastie T, Tibshirani R. Generalized Additive Models. Chapman & Hall: London, 1990.
22. Agresti A. Categorical Data Analysis. Wiley series: United States of America, 2002.
23. Yee T, Wild C. Vector generalized additive models. Journal of the Royal Statistical Society Series B (Methodological) 1996; 58(3):481-493.
24. Yee T. The vgam package for categorical data analysis. Journal of Statistical Software 2010; 32(10):1-34.
25. Kaijser J, Bourne T, Valentin L, Sayasneh A, Van Holsbeke C, Vergote I, Testa A, Franchi D, Van Calster B, Timmerman D. Improving strategies for diagnosing ovarian cancer: a summary of the international ovarian tumor analysis (iota) studies. Ultrasound in Obstetrics & Gynecology 2012; 41(1):9-20. DOI: 10.1002/uog.12323.
26. Timmerman D, Valentin L, Bourne T, Collins W, Verrelst H, Vergote I. Terms, definitions and measurements to describe the sonographic features of adnexal tumors: a consensus opinion from the international ovarian tumor analysis (iota) group. Ultrasound in Obstetrics & Gynecology 2000; 16:500-505. DOI: 10.1046/j.1469-0705.2000.00287.x.
27. Timmerman D, Testa A, Bourne T, Ferrazzi E, Ameye L, Konstantinovic M, Van Calster B, Collins W, Vergote I, Van Huffel S, Valentin L. Logistic regression model to distinguish between the benign and malignant adnexal mass before surgery: a multicenter study by the international ovarian tumor analysis group. Journal of Clinical Oncology 2005; 23:8794-8801. DOI: 10.1200/JCO.2005.01.7632.
28. Vergouwe Y, Steyerberg EW, Foster RS, Sleijfer DT, Fossa SD, Gerl A, de Wit R, Roberts JT, Habbema JDF. Predicting retroperitoneal histology in postchemotherapy testicular germ cell cancer: a model update and multicentre validation with more than 1000 patients. European Urology 2007; 51:424-432. DOI: 10.1016/j.eururo.2006.06.047.
29. König I, Malley J, Weimar C, Diener HC, Ziegler A. Practical experiences on the necessity of external validation. Statistics in Medicine 2007; 26:5499-5511. DOI: 10.1002/sim.3069.
30. Pencina M, D'Agostino R, Steyerberg E. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Statistics in Medicine 2011; 30:11-21. DOI: 10.1002/sim.4085.
31. Harrison DA, Brady AR, Parry GJ, Carpenter JR, Kathy R. Recalibration of risk prediction models in a large multicenter cohort of admissions to adult, general critical care units in the united kingdom. Critical Care Medicine 2006; 34(5):1378-1388. DOI: 10.1097/01.CCM.0000216702.94014.75.
32. Coppus S, van der Veen F, Opmeer B, Mol B, Bossuyt P. Evaluating prediction models in reproductive medicine. Human Reproduction 2009; 24(8):1774-1778. DOI: 10.1093/humrep/dep109.
33. Copas JB. Regression prediction and shrinkage. Journal of the Royal Statistical Society. Series B (Methodological) 1983; 45(3):311-354. DOI: 10.2307/2345402.
34. Harrell FEJ, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine 1996; 15:361-387. DOI: 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4.
35. Steyerberg EW, Borsboom GJJM, van Houwelingen HC, Eijkemans MJC, Habbema JDF. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Statistics in Medicine 2004; 23:2567-2586. DOI: 10.1002/sim.1844.
36. Van Hoorde K, Vergouwe Y, Timmerman D, Van Huffel S, Steyerberg E, Van Calster B. Simple dichotomous updating methods improved the validity of polytomous models. Journal of Clinical Epidemiology 2013; 66:1158-1165. DOI: 10.1016/j.jclinepi.2013.04.014.
37. Ivanov J, Tu JV, Naylor CD. Ready-made, recalibrated, or remodeled?: Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery. Circulation-Journal of the American Heart Association 1999; 99:2098-2104. DOI: 10.1161/01.CIR.99.16.2098.
38. van Houwelingen HC, Rebora P, Reilly M. Validation, calibration and refitting of a familial breast cancer model in sisterships: a case study in the swedish sisters data. Statistics in Medicine 2013; 32:3486-3500. DOI: 10.1002/sim.5781.
39. Toll D, Janssen K, Vergouwe Y, Validation MK. updating and impact of clinical prediction rules: A review. Journal of Clinical Epidemiology 2008; 61:1085-1094. DOI: 10.1016/j.jclinepi.2008.04.008.
40. Moons K, Altman D, Vergouwe Y, Royston P. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. British Medical Journal 2009; 338:b606, DOI: 10.1136/bmj.b606.
41. Vickers AJ, Cronin AM. Everything you always wanted to know about evaluating prediction models (but were too afraid to ask). Urology 2010; 76(6):1298-1301. DOI: 10.1016/j.urology.2010.06.019.
42. Vergouwe Y, Moons KG, Steyerberg EW. External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. American Journal of Epidemiology 2010; 172:971-980. DOI: 10.1093/aje/kwq223.
43. Golub GHG, Heath M, Wahba G. Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 1979; 21(2):215-223. DOI: 10.1080/00401706.1979.10489751.