A spline-based tool to assess and visualize the calibration of multiclass risk predictions

Journal of Biomedical Informatics

Volumn 54, Issue , 2015, Pages 283-293

  Van Hoorde, K ^a   Van Huffel, S ^a   Timmerman, D ^a,b   Bourne, T ^a,b,c   Van Calster, B ^a  

^a UNIVERSITY OF LEUVEN   (Belgium)

^b UNIVERSITY HOSPITALS LEUVEN   (Belgium)

^c QUEEN CHARLOTTE S AND CHELSEA HOSPITAL   (United Kingdom)

Author keywords

Calibration; Logistic regression; Machine learning; Multiclass; Probability estimation; Risk models

Indexed keywords

CALIBRATION; CLASSIFIERS; DECISION TREES; DISEASES; LEARNING SYSTEMS; LOGISTIC REGRESSION; MEDICAL APPLICATIONS; NEAREST NEIGHBOR SEARCH; RANDOM FORESTS; RISK PERCEPTION; SPLINES; SUPPORT VECTOR MACHINES; SUPPORT VECTOR REGRESSION; TUMORS;

K-NEAREST NEIGHBORS; METASTATIC CANCERS; MULTICLASS; MULTINOMIAL LOGISTIC REGRESSION MODELS; OBSERVED PROBABILITIES; PROBABILITY ESTIMATION; RISK MODEL; RISK PREDICTIONS;

RISK ASSESSMENT;

ADULT; ARTICLE; BAYESIAN LEARNING; CALIBRATION; CANCER STAGING; FEMALE; HUMAN; K NEAREST NEIGHBOR; LOGISTIC REGRESSION ANALYSIS; MACHINE LEARNING; MAJOR CLINICAL STUDY; METASTASIS; NEAREST SHRUNKEN CENTROID; OVARY CANCER; OVARY TUMOR; PREDICTION; PRIORITY JOURNAL; RANDOM FOREST; SUPPORT VECTOR MACHINE; AGED; ALGORITHM; CLASSIFICATION; DECISION SUPPORT SYSTEM; MIDDLE AGED; PROCEDURES; RISK ASSESSMENT; STATISTICAL MODEL;

ADULT; AGED; ALGORITHMS; DECISION SUPPORT SYSTEMS, CLINICAL; FEMALE; HUMANS; LOGISTIC MODELS; MACHINE LEARNING; MIDDLE AGED; MODELS, STATISTICAL; OVARIAN NEOPLASMS; RISK ASSESSMENT;

EID: 84927977915     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2014.12.016     Document Type: Article

References (59)

1. Steyerberg E.W. Clinical prediction models: a practical approach to development, validation, and updating 2009, Springer, United States of America
2. König I., Malley J., Weimar C., Diener H.C., Ziegler A. Practical experiences on the necessity of external validation. Stat Med 2007, 26:5499-5511. 10.1002/sim.3069.
3. Altman D., Vergouwe Y., Royston P., Moons K. Prognosis and prognostic research: validating a prognostic model. BMJ 2009, 338:b605. 10.1136/bmj.b605.
4. Peek N., Abu-Hanna A. Clinical prognostic methods: trends and developments. J Biomed Inform 2014, 48:1-4. 10.1016/j.jbi.2014.02.016.
5. Toll D., Janssen K., Vergouwe Y., Moons K. Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol 2008, 61:1085-1094. 10.1016/j.jclinepi.2008.04.008.
6. Matheny M.E., Ohno-Machado L., Resnic F.S. Discrimination and calibration of mortality risk prediction models in interventional cardiology. J Biomed Inform 2005, 38(5):367-375
7. Van Calster B., Vickers A.J. Calibration of risk prediction models impact on decision-analytic performance. Med Decis Making 2014, 0272989X14547233
8. Janes H., Pepe M.S., Huang Y. A framework for evaluating markers used to select patient treatment. Med Decis Making 2014, 34(2):159-167
9. Helfand M. Shared decision making, decision aids, and risk communication. Med Decis Making 2007
10. Brinkman S., Abu-Hanna A., van der Veen A., de Jonge E., de Keizer N.F. A comparison of the performance of a model based on administrative data and a model based on clinical data: effect of severity of illness on standardized mortality ratios of intensive care units Crit Care Med 2012, 40(2):373-378. 10.1097/CCM.0b013e318232d7b0.
11. Simon R. The use of genomics in clinical trial design. Clin Cancer Res 2008, 14(19):5984-5993
12. Pauker S.G., Kassirer J.P. Therapeutic decision making: a cost-benefit analysis. N Engl J Med 1975, 293(5):229-234
13. Taktak A.F., Eleuteri A., Lake S.P., Fisher A.C. A web-based tool for the assessment of discrimination and calibration properties of prognostic models. Comput Biol Med 2008, 38:785-791. 10.1016/j.compbiomed.2008.04.005.
14. Austin P.C., Steyerberg E.W. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med 2014, 33(3):517-535. 10.1002/sim.5941.
15. Van Hoorde K., Vergouwe Y., Timmerman D., Van Huffel S., Steyerberg E.W., Van Calster B. Assessing calibration of multinomial risk prediction models. Stat Med 2014, 33(15):2585-2596. 10.1002/sim.6114.
16. Condous G., Van Calster B., Kirk E., Haider Z., Timmerman D., Van Huffel S., et al. Prediction of ectopic pregnancy in women with a pregnancy of unknown location. Ultrasound Obstet Gynecol 2007, 29(6):680-687. 10.1002/uog.4015.
17. Djavan B., Remzi M., Zlotta A., Seitz C., Snow P., Marberger M. Novel artificial neural network for early detection of prostate cancer. J Clin Oncol 2002,20(4):921-929
18. Kattan M.W. Comparison of cox regression with other methods for determining 10. prediction models and nomograms. J Urol 2003, 170(6):S6-S10. 10.1097/01.ju.0000094764.56269.2d
19. Van Esbroeck A., Rubinfeld I., Hall B., Syed Z. Quantifying surgical complexity with machine learning: looking beyond patient factors to improve surgical models. Surgery 2014, 10.1016/j.surg.2014.04.034.
20. Klement R.J., Allgäuer M., Appold S., Dieckmann K., Ernst I., Ganswindt U., et al. Support vector machine-based prediction of local tumor control after stereotactic body radiation therapy for early-stage non-small cell lung cancer. Int J Radiat Oncol Biol Phys 2014, 88(3):732-738. 10.1016/j.ijrobp.2013.11.216.
21. Lisboa P.J., Taktak A.F. The use of artificial neural networks in decision support in cancer: a systematic review. Neural Netw 2006, 19(4):408-415. 10.1016/j.neunet.2005.10.007.
22. Cruz J.A., Wishart D.S. Applications of machine learning in cancer prediction and prognosis. Cancer Inform 2006, 2:59
23. Sajda P. Machine learning for detection and diagnosis of disease. Ann Rev Biomed Eng 2006, 8:537-565. 10.1146/annurev.bioeng.8.061505.095802.
24. Kruppa J., Ziegler A., König I.R. Risk estimation and risk prediction using machine-learning methods. Hum Genet 2012, 131(10):1639-1654. 10.1007/s00439-012-1194-y.
25. Van Calster B, Luts J, Suykens JA, Condous G, Bourne T, Timmerman D, et al. Comparing methods for multi-class probabilities in medical decision making using LS-SVMs and kernel logistic regression. In: Marques de Sá J, Alexandre L, Duch W, Mandic D, editors. Artificial neural networks - ICANN 2007 lecture notes in computer science, p. 139-48
26. Kruppa J., Liu Y., Biau G., Kohler M., König I.R., Malley J.D., et al. Probability estimation with machine learning methods for dichotomous and multi-category outcome: theory. Biom J 2014, 56(4):534-563. 10.1002/bimj.201300068.
27. Kruppa J., Liu Y., Diener H.C., Holste T., Weimar C., König I.R., et al. Probability estimation with machine learning methods for dichotomous and multicategory outcome: applications. Biom J 2014, 56(4):564-583. 10.1002/bimj.201300077.
28. Zadrozny B., Elkan C. Transforming classifier scores into accurate multiclass probability estimates. Proceedings of the eight international conference on knowledge discovery and data mining 2002, 694-699. ACM Press, New York
29. Malley J., Kruppa J., Dasgupta A., Malley K., Ziegler A. Probability machines: consistent probability estimation using nonparametric learning machines. Methods Inform Med 2012, 51(1):74. 10.3414/ME00-01-0052.
30. Jiang X., Osl M., Kim J., Ohno-Machado L. Calibrating predictive model estimates to support personalized medicine. J Am Med Inform Assoc 2012, 19(2):263-274. 10.1136/amiajnl-2011-000291.
31. Matheny M.E., Resnic F.S., Arora N., Ohno-Machado L. Effects of SVM parameter optimization on discrimination and calibration for post-procedural PCI mortality. J Biomed Inform 2007, 40(6):688-697. 10.1016/j.jbi.2007.05.008.
32. Jiang X., Menon A., Wang S., Kim J., Ohno-Machado L. Doubly optimized calibrated support vector machine (DOC-SVM): an algorithm for joint optimization of discrimination and calibration. PloS One 2012, 7(11):e48823. 10.1371/journal.pone.0048823.
33. Agresti A. Categorical data analysis. Wiley series, United States of America; 2002
34. Yee T., Wild C. Vector generalized additive models. J Roy Stat Soc Ser B (Methodol) 1996, 58(3):481-493
35. Yee T.W. Vector splines and other vector smoothers. COMPSTAT: proceedings in computational statistics 2000, 529-534. Springer. J.G. Betlehem, P.G.M. van der Heijden (Eds.)
36. Steyerberg E.W., Vickers A.J., Cook N.R., Gerds T., Gonen M., Obuchowski N., et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010, 21(1):128-138. 10.1097/EDE.0b013e3181c30fb2.
37. Timmerman D., Testa A., Bourne T., Ferrazzi E., Ameye L., Konstantinovic M., et al. Logistic regression model to distinguish between the benign and malignant adnexal mass before surgery: a multicenter study by the international ovarian tumor analysis group. J Clin Oncol 2005, 23:8794-8801. 10.1200/JCO.2005.01.7632.
38. Timmerman D., Van Calster B., Testa A., Guerriero S., Fischerova D., Lissoni A., et al. Ovarian cancer prediction in adnexal masses using ultrasound-based logistic regression models: a temporal and external validation study by the iota group. Ultrasound Obstet Gynecol 2010, 36:226-234. 10.1002/uog.7636.
39. Van Holsbeke C., Van Calster B., Testa A., Domali E., Lu C., Van Huffel S., et al. Prospective interval validation of mathematical models to predict malignancy in adnexal masses: results from the international ovarian tumor analysis study. Clin Cancer Res 2009, 15:648-691. 10.1158/1078-0432.CCR-08-0113.
40. Van Calster B., Van Hoorde K., Valentin L., Testa A.C., Fischerova D., Van Holsbeke C., et al. Evaluating the risk of ovarian cancer prior to surgery using the ADNEX risk model: diagnostic study to differentiate between benign, borderline, stage i invasive, advanced stage invasive, and secondary metastatic tumours. BMJ 2014, 349:g5920. 10.1136/bmj.g5920.
41. Schenker N., Taylor J. Partially parametric techniques for multiple imputation. Comput Stat Data Anal 1996, 22:425-446. 10.1016/0167-9473(95)00057-7.
42. Sterne J., White I., Carlin J., Spratt M., Royston P., Kenward M., et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ 2009, 338:b2393. 10.1136/bmj.b2393.
43. Lasko T.A., Bhagwat J.G., Zou K.H., Ohno-Machado L. The use of receiver operating characteristic curves in biomedical informatics. J Biomed Inform 2005, 38(5):404-415
44. Van Calster B., Vergouwe Y., Van Belle V., Looman C., Timmerman D., Steyerberg E. Assessing the discriminative ability of risk models for more than two outcome -categories: a perspective. Eur J Epidemiol 2012, 27(10):761-770.10.1007/s10654-0129733-3
45. Hand D.J., Till R.J. A simple generalization of the area under the ROC curve for multiple class classification problems. Mach Learn 2001, 45:171-186. 10.1023/A:1010920819831.
46. Van Calster B., Van Belle V., Vergouwe Y., Timmerman D., Van Huffel S., Steyerberg E.W. Extending the c statistic to nominal polytomous outcomes: the polytomous discrimination index. Stat Med 2012, 31:2610-2626. 10.1002/sim.5321.
47. Schölkopf B., Smola A.J. Learning with kernels: support vector machines, regularization, optimization, and beyond 2002, MIT Press
48. Steinbach M., Tan P.N. kNN: k-nearest neighbors. The top ten algorithms in data mining 2009, 151-162. Chapman & Hall/CRC. X. Wu, V. Kumar (Eds.)
49. Breiman L. Random forests. Mach Learn 2001, 45(1):5-32. 10.1023/A:1010933404324.
50. Hand D.J. Naïve Bayes. The top ten algorithms in data mining 2009, 163-178. Chapman & Hall/CRC. X. Wu, V. Kumar (Eds.)
51. Tibshirani R., Hastie T., Narasimhan B., Chu G. Class prediction by nearest shrunken centroids, with applications to DNA microarrays. Stat Sci 2003, 18(1):104-117
52. Frank E, Kramer S. Ensembles of nested dichotomies for multi-class problems. In: Proceedings of the 21st international conference on machine learning. Banff, Canada, 39
53. Hastie T., Tibshirani R. Classification by pairwise coupling. Ann Stat 1998, 26(2):451-471. 10.1214/aos/1028144844.
54. Wu T., Lin C., Weng R. Probability estimates for multi-class classification by pairwise coupling. J Mach Learn Res 2004, 5:975-1005
55. Rish I. An empirical study of the naive Bayes classifier. In: IJCAI 2001 workshop on empirical methods in artificial intelligence, vol. 3. p. 41-6
56. Harrell F. Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis 2001, Springer, UK
57. Hosmer D., Hosmer T., Le Cessie S., Lemeshow S. A comparison of goodness-of-fit tests for the logistic regression models. Stat Med 1997, 16:965-980. 10.1002/(SICI)1097-0258(19970515)
58. Steyerberg E.W., van der Ploeg T., Van Calster B. Risk prediction with machine learning and regression methods. Biom J 2014, 56(4):601-606. 10.1002/bimj.201300297.
59. Golub G.H.G., Heath M., Wahba G. Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 1979, 21(2):215-223. 10.1080/00401706.1979.10489751.