Optimal thresholds by maximizing or minimizing various metrics via ROC-type analysis

Academic Radiology

Volumn 20, Issue 7, 2013, Pages 807-815

  Zou, Kelly H ^a   Yu, Ching Ray ^a   Liu, Kezhen ^b   Carlsson, Martin O ^a   Cabrera, Javier ^b  

^a PFIZER INC   (United States)

^b RUTGERS UNIVERSITY   (United States)

Author keywords

Optimal threshold; Receiver operating characteristic analysis; Sensitivity; Specificity

Indexed keywords

ARTICLE; BOOTSTRAPPING; CLINICAL DECISION MAKING; EUCLIDIAN DISTANCE; FLANK PAIN; KAPPA STATISTICS; MONTE CARLO METHOD; MUTUAL INFORMATION; PARAMETERS; PERCENT OF CORRECT DIAGNOSIS; PRIORITY JOURNAL; RADIODIAGNOSIS; RECEIVER OPERATING CHARACTERISTIC; SPIRAL COMPUTER ASSISTED TOMOGRAPHY; STATISTICAL ANALYSIS; TREATMENT OUTCOME; URETER STONE; YOUDEN INDEX;

HUMANS; MODELS, STATISTICAL; ROC CURVE; SENSITIVITY AND SPECIFICITY; TOMOGRAPHY, SPIRAL COMPUTED; URETER; URETERAL CALCULI;

EID: 84878929770     PISSN: 10766332     EISSN: 18784046     Source Type: Journal    
DOI: 10.1016/j.acra.2013.02.004     Document Type: Article

References (42)

1. Liu A., Schisterman E.F., Zhu Y. On linear combinations of biomarkers to improve diagnostic accuracy. Stat Med 2005, 24:37-47.
2. Perkins N.J., Schisterman E.F., Vexler A. Receiver operating characteristic curve inference from a sample with a limit of detection. Am J Epidemiol 2007, 165:325-333.
3. Perkins N.J., Schisterman E.F., Vexler A. Generalized ROC curve inference for a biomarker subject to a limit of detection and measurement error. Stat Med 2009, 28:1841-1860.
4. Woodcock J. Assessing the clinical utility of diagnostics used in drug therapy. Clin Pharmacol Ther 2010, 88:765-773.
5. Wagner J.A., Wright E.C., Ennis M.M., et al. Utility of adiponectin as a biomarker predictive of glycemic efficacy is demonstrated by collaborative pooling of data from clinical trials conducted by multiple sponsors. Clin Pharmacol Ther 2009, 86:619-625.
6. Biomarkers and surrogate endpoints preferred definitions and conceptual framework. Clin Pharmacol Ther 2001, 69:89-95. Biomarkers Definitions Working Group.
7. Lesko L.J., Atkinson A.J. Use of biomarkers and surrogate endpoints in drug development and regulatory decision making: criteria, validation, strategies. Annu Rev Pharmacol Toxicol 2001, 41:347-366.
8. Metz C.E. Basic principles of ROC analysis. Semin Nucl Med 1978, 8:283-298.
9. Youden W.J. Index for rating diagnostic tests. Cancer 1950, 3:32-35.
10. Fluss R., Faraggi D., Reiser B. Estimation of the Youden Index and its associated cutoff point. Biomed J 2005, 47:458-472.
11. Perkins N.J., Schisterman E.F. The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. Am J Epidemiol 2006, 163:670-675.
12. Schisterman E.F., Faraggi D., Reiser B., et al. Youden Index and the optimal threshold for markers with mass at zero. Stat Med 2008, 27:297-315.
13. Nakas C.T., Alonzo T.A., Yiannoutsos C.T. Accuracy and cut-off point selection in three-class classification problems using a generalization of the Youden index. Stat Med 2010, 29:2946-2955.
14. ® 2007, SAS Institute Inc, Cary, NC.
15. Zou K.H., Wells W.M., Kikinis R., et al. Three validation metrics for automated probabilistic image segmentation of brain tumours. Stat Med 2004, 23:1259-1282.
16. Davila M., Christenson L.J., Sontheimer R.D. Epidemiology and outcomes ofdermatology in-patient consultations in a Midwestern U.S. university hospital. Dermatol Online J 2010, 16:12.
17. Fielding J.R., Silverman S.G., Samuel S., et al. Unenhanced helical CT of ureteral stones: a replacement for excretory urography in planning treatment. AJR Am J Roentgenol 1998, 171:1051-1053.
18. Zou K.H., Tempany C.M., Fielding J.R., et al. Original smooth receiver operating characteristic curve estimation from continuous data: statistical methods for analyzing the predictive value of spiral CT of ureteral stones. Acad Radiol 1998, 5:680-687.
19. O'Malley A.J., Zou K.H., Fielding J.R., et al. Bayesian regression methodology for estimating a receiver operating characteristic curve with two radiologic applications: prostate biopsy and spiral CT of ureteral stones. Acad Radiol 2001, 8:713-725.
20. McClish D.K. Evaluation of the accuracy of medical tests in a region around the optimal point. Acad Radiol 2012, 19:1484-1490.
21. Eng J. Teaching receiver operating characteristic analysis: an interactive laboratory exercise. Acad Radiol 2012, 19:1452-1456.
22. Dorfman D.D., Alf E. Maximum-likelihood estimation of parameters ofsignal-detection theory and determination of confidence intervals - rating-method data. J Math Psychol 1969, 6:487-496.
23. Metz C.E., Herman B.A., Shen J.H. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Stat Med 1998, 17:1033-1053.
24. Pesce L.L., Horsch K., Drukker K., et al. Semiparametric estimation of the relationship between ROC operating points and the test-result scale: application to the proper binormal model. Acad Radiol 2011, 18:1537-1548.
25. Alemayehu D., Zou K.H. Applications of ROC analysis in medical research: recent developments and future directions. Acad Radiol 2012, 19:1457-1464.
26. Metz C.E. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol 1989, 24:234-245.
27. Eng J. ROC analysis: Web-based calculator for ROC curves. Available at: Accessed December 12,2012. http://www.jrocfit.org.
28. Box G.E.P., Cox D.R. An analysis of transformations. JRSSB 1964, 26:211-252.
29. Zou K.H., O'Malley A.J. A Bayesian hierarchical non-linear regression modelinreceiver operating characteristic analysis of clustered continuous diagnostic data. Biomed J 2005, 47:417-427.
30. O'Malley A.J., Zou K.H. Bayesian multivariate hierarchical transformation models for ROC analysis. Stat Med 2006, 25:459-479.
31. Zou K.H., Carlsson M.O., Yu C.R. Comparison of adjustment methods for stratified two-sample tests in the context of ROC analysis. Biomed J 2012, 54:249-263.
32. Zou K.H., Warfield S.K., Fielding J.R., et al. Statistical validation based onparametric receiver operating characteristic analysis of continuous classification data. Acad Radiol 2003, 10:1359-1368.
33. Hanley J.A. Receiver operating characteristic (ROC) analysis: the state of the art. Crit Revi Diagnostic Imaging 1989, 29:307-335.
34. Hanley J.A., McNeil B.J. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982, 143:29-36.
35. Wieand S., Gail M.H., James B.R., et al. A family of nonparametric statistics for comparing diagnostic makers with paired or unpaired data. Biometrika 1989, 76:585-592.
36. Zou K.H., Liu A., Bandos A.I., et al. Statistical evaluation of diagnostic performance: topics in ROC analysis 2011, Chapman & Hall/CRC Press, Boca Raton, FL.
37. Efron B., Tibshirani R.J. Introduction to the bootstrap 1994, Chapman & Hall/CRC Press, Boca Raton, FL.
38. R: a language and environment for statistical computing 2008, R Development Core Team, R Foundation for Statistical Computing, Vienna, Austria, Available at:, Accessed December 12, 2012. http://www.R-project.org.
39. Robert C.P., Casella G. Monte Carlo statistical methods 2010, Springer Verlag, New York, NY.
40. Zhou X.H., Obuchowski N.A., McClish D.K. Statistical methods in diagnostic medicine 2002, Wiley & Sons Inc, New York, NY.
41. Pepe M.S. The statistical evaluation of medical tests for classification andprediction 2003, Oxford University Press, Oxford, UK.
42. Suss R. Sensitivity and specificity: alien edition: a light-hearted look at statistics. Can Fam Physicians 2007, 53:1743-1744.