Calibration of medical diagnostic classifier scores to the probability of disease

Statistical Methods in Medical Research

Volumn 27, Issue 5, 2018, Pages 1394-1409

  Chen, Weijie ^a   Sahiner, Berkman ^a   Samuelson, Frank ^a   Pezeshk, Aria ^a   Petrick, Nicholas ^a  

^a CENTER FOR DRUG EVALUATION AND RESEARCH   (United States)

Author keywords

Calibration; classifier; probability of disease; rationality

Indexed keywords

CALIBRATION; CLASSIFICATION; CONFIDENCE INTERVAL; DIAGNOSIS; DISEASES; HUMAN; NONPARAMETRIC TEST; PROBABILITY; SAMPLE SIZE; STATISTICS;

CALIBRATION; CONFIDENCE INTERVALS; DIAGNOSIS; DISEASE; HUMANS; PROBABILITY; SAMPLE SIZE; STATISTICS AS TOPIC; STATISTICS, NONPARAMETRIC;

EID: 85044352117     PISSN: 09622802     EISSN: 14770334     Source Type: Journal    
DOI: 10.1177/0962280216661371     Document Type: Article

References (40)

1. Doi K. Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Computer Med Imag Graph 2007; 31: 198–211.
2. Shi L, Campbell G, Jones WD, Consortium MAQC-II. The MAQC-II project: a comprehensive study of common practices for the development and validation of microarray-based predictive models. Nat Biotechnol 2010; 28: 827–838.
3. Gail MH, Brinton LA, Byar DP, Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. J Natl Canc Inst 1989; 81: 1879–1886.
4. Jain A, Nandakumar K, Ross A. Score normalization in multimodal biometric systems. Pattern Recogn 2005; 38: 2270–2285.
5. Platt JC, Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Smola AJ, Bartlett PJ, Schölkopf B, (eds). Advances in large margin classifiers, Cambridge, MA: MIT Press, 2000, pp. 61–74.
6. Zadrozny B and Elkan C. Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the eighth ACM SIGKDD international conference on knowledge discovery and data mining, 2002, pp.694–699. New York: ACM
7. Ayer M, Brunk HD, Ewing GM, An empirical distribution function for sampling with incomplete information. Ann Math Stat 1955; 26: 641–647.
8. Niculescu-Mizil A and Caruana R. Predicting good probabilities with supervised learning. In: Proceedings of the 22nd international conference on machine learning, ICML’05, 2005, pp. 625–632. New York: ACM
9. Pesce LL, Horsch K, Drukker K, Semiparametric estimation of the relationship between ROC operating points and the test-result scale: application to the proper binormal model. Acad Radiol 2011; 18: 1537–1548.
10. Metz CE, Pan X. Proper binormal ROC curves: theory and maximum-likelihood estimation. J Math Psychol 1999; 1: 1–33.
11. Chen W, Sahiner B, Samuelson F, Investigation of methods for calibration of classifier scores to probability of disease. In: Mello-Thoms CR, Kupinski MA, (eds). Medical imaging 2015: image perception, observer performance, and technology assessment 21 February 2015; Vol. 9416, Orlando, Florida, United States: SPIE, pp. 94161E–8. http://dx.doi.org/10.1117/12.2082142.
12. Schmid C, Griffith J, Multivariate classification rules: calibration and discrimination. In: Armitage P, Colton T, (eds). Encyclopedia of biostatistics, Chichester: Wiley, 2005, pp. 3491–3497.
13. Bamber D. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psych 1975; 12: 387–415.
14. Diamond GA. What price perfection? Calibration and discrimination of clinical prediction models. J Clin Epidemiol 1992; 45: 85–89.
15. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2003; 10: 670–672.
16. Su JQ, Liu JS. Linear combinations of multiple diagnostic markers. J Am Stat Assoc 1993; 88: 1350–1355.
17. Egan JP. Signal detection theory and ROC analysis, New York: Academic Press, 1975.
18. Press WH, Teukolsky SA, Vetterling WT, Numerical recipes in C: the art of scientific computing, 2nd ed. New York: Cambridge University Press, 1992.
19. Lin HT, Lin CJ, Weng R. A note on platt’s probabilistic outputs for support vector machines. Mach Learn 2007; 68: 267–276.
20. Casella G, Berger RL. Statistical inference. 2nd ed. (Duxbury advanced series, Australia), Pacific Grove, CA: Duxbury/Thomson Learning, 2002.
21. Metz CE, Herman BA, Shen J. Maximum-likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Stat Med 1998; 17: 1033–1053.
22. Efron B, Tibshirani RJ. An introduction to the bootstrap: monographs on statistics and applied probability, New York: Chapman & Hall, 1993.
23. Lloyd CJ. Estimation of a convex ROC curve. Stat Prob Lett 2002; 59: 99–99.
24. Fawcett T, Niculescu-Mizil A. PAV and the ROC convex hull. Mach Learn 2007; 68: 97–106.
25. Zhou XH, Obuchowski NA, McClish DK. Statistical methods in diagnostic medicine, Hoboken, NJ: Wiley & Sons, 2002.
26. Pepe MS. The statistical evaluation of medical tests for classification and prediction, London: Oxford University Press, 2003.
27. Steyerberg EW, Vickers AJ, Cook NR, Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21: 128–138.
28. Hilden J, Gerds TA. A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index. Stat Med 2014; 33: 3405–3414.
29. Leening MJ, Vedder MM, Witteman JC, Net reclassification improvement: computation, interpretation, and controversies a literature review and clinician’s guide. Ann Intern Med 2014; 160: 122–131.
30. Pepe MS, Fan J, Feng Z, The net reclassification index (NRI): a misleading measure of prediction improvement even with independent test data sets. Stat Biosci 2015; 7: 282–295.
31. Brier GW. Verification of forecasts expressed in terms of probability. Mon Weather Rev 1950; 78: 1–3.
32. Mossman D, Peng H. Using dual beta distributions to create “proper” ROC curves based on rating category data. Med Decis Making 2016; 36: 349–365.
33. Fukunaga K, Hayes RR. Effects of sample size in classifier design. IEEE Trans Pattern Anal Mach Intell 1989; 11: 873–885.
34. Cover TM. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Trans Electron Comput 1965; 14: 326–334.
35. Hastie T, Tibshirani R, Friedman J. The elements of statistical learning: data mining, inference, and prediction, New York: Springer, 2001.
36. Horsch K, Giger ML, Metz CE. Prevalence scaling: applications to an intelligent workstation for the diagnosis of breast cancer. Acad Radiol 2008; 15: 1446–1457.
37. Rasmussen CE, Williams CKI. Gaussian processes for machine learning (adaptive computation and machine learning), Cambridge, MA: The MIT Press, 2005.
38. MacKay DJ. A practical Bayesian framework for backpropagation networks. Neural Comput 1992; 4: 448–472.
39. Horsch K, Pesce LL, Giger ML, A scaling transformation for classifier output based on likelihood ratio: applications to a cad workstation for diagnosis of breast cancer. Med Phys 2012; 39: 2787–2804.
40. Kondratovich MV and Yousef WA. Evaluation of accuracy and optimal cutoff of diagnostic devices in the same study. In: Proceedings of the 2005 joint statistical meetings, section on statistics in epidemiology, pp.2547–2551. Alexandria, VA: American Statistical Association