Diagnostic accuracy and receiver-operating characteristics curve analysis in surgical research and decision making

Annals of Surgery

Volumn 253, Issue 1, 2011, Pages 27-34

  Søreide, Kjetil ^a   Kørner, Hartwig ^b   Søreide, Jon Arne ^b  

^a STAVANGER UNIVERSITY HOSPITAL   (Norway)

^b NONE

Author keywords

[No Author keywords available]

Indexed keywords

BIOLOGICAL MARKER; C REACTIVE PROTEIN; CARCINOEMBRYONIC ANTIGEN; ESTROGEN RECEPTOR; PROGESTERONE RECEPTOR; PROSTATE SPECIFIC ANTIGEN; TUMOR MARKER;

ANASTOMOSIS LEAKAGE; ANEURYSM SURGERY; AORTA ANEURYSM; APPENDICITIS; AREA UNDER THE CURVE; BREAST CANCER; BREAST SURGERY; CANCER ADJUVANT THERAPY; CANCER RESEARCH; CANCER SURGERY; CLINICAL PRACTICE; CLINICAL RESEARCH; COLORECTAL CANCER; COLORECTAL SURGERY; COMPUTER PROGRAM; CUSTOMIZED PROBABILITY INDEX; DIAGNOSTIC ACCURACY; DIAGNOSTIC TEST; FOLLOW UP; FOOD AND DRUG ADMINISTRATION; GLASGOW ANEURYSM SCORE; GOLD STANDARD; HUMAN; INJURY; LIVER SURGERY; MEDICAL DECISION MAKING; MODIFIED CUSTOMIZED PROBABILTY INDEX; OUTCOMES RESEARCH; PREDICTION; PREDICTIVE VALUE; PRIORITY JOURNAL; PROBABILITY; PROGNOSIS; PROSTATE CANCER; RADIOLOGIST; RECEIVER OPERATING CHARACTERISTIC; REVIEW; RISK; SCORING SYSTEM; STATISTICAL ANALYSIS; SURGEON; SURGERY; TREATMENT OUTCOME; VALIDATION STUDY; VALIDITY; VASCULAR PHYSIOLOGY AND OPERATIVE SEVERITY SCORE FOR THE ENUMERATION OF MORTALITY AND MORBIDITY;

DECISION MAKING; DIAGNOSTIC IMAGING; HUMANS; PATIENT SELECTION; PREDICTIVE VALUE OF TESTS; ROC CURVE; SURGICAL PROCEDURES, OPERATIVE;

EID: 78751630983     PISSN: 00034932     EISSN: 15281140     Source Type: Journal    
DOI: 10.1097/SLA.0b013e318204a892     Document Type: Review

References (102)

1. Smellie WS. What is a significant difference between sequential laboratory results? J Clin Pathol. 2008;61:419-425.
2. Smellie WS. When is "abnormal" abnormal? Dealing with the slightly out of range laboratory result. J Clin Pathol. 2006;59:1005-1007.
3. Obuchowski NA. An ROC-type measure of diagnostic accuracy when the gold standard is continuous-scale. Stat Med. 2006;25:481-493.
4. Van Walraven C, Naylor CD. Do we know what inappropriate laboratory utilization is? A systematic review of laboratory clinical audits. JAMA. 1998;280:550-558.
5. Søreide K. Receiver-operating characteristic curve analysis in diagnostic, prognostic and predictive biomarker research. J Clin Pathol. 2009;62:1-5.
6. Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 1993;39:561-577.
7. Leeflang MM, Bossuyt PM, Irwig L. Diagnostic test accuracy may vary with prevalence: implications for evidence-based diagnosis. J Clin Epidemiol. 2009;62:5-12.
8. Lang TA, Secic M. Chapter 10. Determining the Presence or Absence of Disease: reporting the performance Characteritics of Diagnostic tests. In: Lang TA, Secic M, eds. How to report Statistics in Medicine Annotated Guidelines for Authors, Editors, and Reviewers. 2nd ed. Philadelphia: American College of Physicians; 2006:125-49.
9. Zlobec I, Steele R, Terracciano L, et al. Selecting immunohistochemical cut-off scores for novel biomarkers of progression and survival in colorectal cancer. J Clin Pathol. 2007;60:1112-1116.
10. Zou KH, O'Malley AJ, Mauri L. Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation. 2007;115:654-657.
11. Søreide K, Gudlaugsson E, Skaland I, et al. Metachronous cancer development in patients with sporadic colorectal adenomas-multivariate risk model with independent and combined value of hTERT and survivin. Int J Colorectal Dis. 2008;23:389-400.
12. Tang P, Skinner KA, Hicks DG. Molecular classification of breast carcinomas by immunohistochemical analysis: are we ready? Diagn Mol Pathol. 2009;18:125-132.
13. Kørner H, Søreide K, Stokkeland PJ, et al. Diagnostic accuracy of serum-carcinoembryonic antigen in recurrent colorectal cancer: a receiver operating characteristic curve analysis. Ann Surg Oncol. 2007;14:417-423.
14. Gerszten RE, Wang TJ. The search for new cardiovascular biomarkers. Nature. 2008;451:949-952.
15. Baker SG. The central role of receiver operating characteristic (ROC) curves in evaluating tests for the early detection of cancer. J Natl Cancer Inst. 2003;95:511-515.
16. Johnson NP. Advantages to transforming the receiver operating characteristic (ROC) curve into likelihood ratio co-ordinates. Stat Med. 2004;23:2257-2266.
17. The magnificent ROC. http://www.anaesthetist.com/mnm/stats/roc/Findex. htm, 2010. Accessed September 12, 2010.
18. Li CR, Liao CT, Liu JP. A non-inferiority test for diagnostic accuracy based on the paired partial areas under ROC curves. Stat Med. 2008;27:1762-1776.
19. Dodd LE, Pepe MS. Partial AUC estimation and regression. Biometrics. 2003;59:614-623.
20. Altman DG, Bland JM. Diagnostic tests 3: receiver operating characteristic plots. BMJ. 1994;309:188.
21. Swets JA. Detection theory and psychophysics: a review. Psychometrika. 1961;26:49-63.
22. Lusted LB. Signal detectability and medical decision-making. Science. 1971;171:1217-1219.
23. Lusted LB. Decision-making studies in patient management. N Engl J Med. 1971;284:416-424.
24. Obuchowski NA. ROC analysis. AJR Am J Roentgenol. 2005;184:364-372.
25. Obuchowski NA, Lieber ML, Wians FH, Jr. ROC curves in clinical chemistry: uses, misuses, and possible solutions. Clin Chem. 2004;50: 1118-1125.
26. Pepe MS, Janes HE. Gauging the performance of SNPs, biomarkers, and clinical factors for predicting risk of breast cancer. J Natl Cancer Inst. 2008;100:978-979.
27. Walter SD, Sinuff T. Studies reporting ROC curves of diagnostic and prediction data can be incorporated into meta-analyses using corresponding odds ratios. J Clin Epidemiol. 2007;60:530-534.
28. Musial J, Swadzba J, Motyl A, et al. Clinical significance of antiphospholipid protein antibodies. Receiver operating characteristics plot analysis. J Rheuma-tol. 2003;30:723-730.
29. Cheung R, Altschuler MD, D'Amico AV, et al. Using the receiver operating characteristic curve to select pretreatment and pathologic predictors for early and late postprostatectomy PSA failure. Urology. 2001;58:400-405.
30. Kørner H, Søreide JA, Søndenaa K. Diagnostic accuracy of inflammatory markers in patients operated on for suspected acute appendicitis: a receiver operating characteristic curve analysis. Eur J Surg. 1999;165:679-685.
31. Kørner H, Nielsen HJ, Søreide JA, et al. Diagnostic accuracy of C-reactive protein for intraabdominal infections after colorectal resections. J Gastrointest Surg. 2009;13:1599-1606.
32. Houssami N, Ciatto S, Macaskill P, et al. Accuracy and surgical impact of magnetic resonance imaging in breast cancer staging: systematic review and meta-analysis in detection of multifocal and multicentric cancer. J Clin Oncol. 2008;26:3248-3258.
33. Visser JJ, Williams M, Kievit J, et al. Prediction of 30-day mortality after endovascular repair or open surgery in patients with ruptured abdominal aortic aneurysms. J Vasc Surg. 2009;49:1093-1099.
34. Bohm N, Wales L, Dunckley M, et al. Objective risk-scoring systems for repair of abdominal aortic aneurysms: applicability in endovascular repair? Eur J Vasc Endovasc Surg. 2008;36:172-177.
35. van la Parra RF, Ernst MF, Bevilacqua JL, et al. Validation of a nomogram to predict the risk of nonsentinel lymph node metastases in breast cancer patients with a positive sentinel node biopsy: validation of the MSKCC breast nomogram. Ann Surg Oncol. 2009;16:1128-1135.
36. Fernandez-Ranvier GG, Weng J, Yeh RF, et al. Identification of biomarkers of adrenocortical carcinoma using genomewide gene expression profiling. Arch Surg. 2008;143:841-846; discussion 6.
37. Parodi S, Muselli M, Fontana V, et al. ROC curves are a suitable and flexible tool for the analysis of gene expression profiles. Cytogenet Genome Res. 2003;101:90-91.
38. Li J, Fine JP. ROC analysis with multiple classes and multiple tests: methodology and its application in microarray studies. Biostatistics. 2008;9:566-576.
39. Ye J, Liu H, Kirmiz C, et al. On the analysis of glycomics mass spectrometry data via the regularized area under the ROC curve. BMC Bioinformatics. 2007;8:477.
40. Pepe MS, Cai T, Longton G. Combining predictors for classification using the area under the receiver operating characteristic curve. Biometrics. 2006;62:221-229.
41. Janes H, Pepe M. The optimal ratio of cases to controls for estimating the classification accuracy of a biomarker. Biostatistics. 2006;7:456-468.
42. Kester AD, Buntinx F. Meta-analysis of ROC curves. Med Decis Making. 2000;20:430-439.
43. He X, Frey EC. The validity of three-class Hotelling trace (3-HT) in describing three-class task performance: comparison of three-class volume under ROC surface (VUS) and 3-HT. IEEE Trans Med Imaging. 2009;28:185-193.
44. He X, Frey EC. The meaning and use of the volume under a three-class ROC surface (VUS). IEEE Trans Med Imaging. 2008;27:577-588.
45. He X, Frey E. ROC, LROC, FROC, AFROC: an alphabet soup. J Am Coll Radiol. 2009;6:652-655.
46. Liu JP, Ma MC, Wu CY, et al. Tests of equivalence and non-inferiority for diagnostic accuracy based on the paired areas under ROC curves. Stat Med. 2006;25:1219-1238.
47. He X, Metz CE, Tsui BM, et al. Three-class ROC analysis-a decision theoretic approach under the ideal observer framework. IEEE Trans Med Imaging. 2006;25:571-581.
48. Zheng Y, Heagerty PJ. Semiparametric estimation of time-dependent ROC curves for longitudinal marker data. Biostatistics. 2004;5:615-632.
49. Cai T, Moskowitz CS. Semi-parametric estimation of the binormal ROC curve for a continuous diagnostic test. Biostatistics. 2004;5:573-586.
50. Alonzo TA, Pepe MS. Distribution-free ROC analysis using binary regression techniques. Biostatistics. 2002;3:421-432.
51. Reiser B. Measuring the effectiveness of diagnostic markers in the presence of measurement error through the use of ROC curves. Stat Med. 2000;19:2115-2129.
52. Faraggi D. The effect of random measurement error on receiver operating characteristic (ROC) curves. Stat Med. 2000;19:61-70.
53. Metz CE, Pan X. "Proper" Binormal ROC Curves: Theory and Maximum-Likelihood Estimation. J Math Psychol. 1999;43:1-33.
54. Zou KH, Hall WJ, Shapiro DE. Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests. Stat Med. 1997;16:2143-2156.
55. Hanley JA. Receiver operating characteristic (ROC) methodology: the state of the art. Crit Rev Diagn Imaging. 1989;29:307-335.
56. Clark RD, Webster-Clark DJ. Managing bias in ROC curves. J Comput Aided Mol Des. 2008;22:141-146.
57. Ransohoff DF. Bias as a threat to the validity of cancer molecular-marker research. Nat Rev Cancer. 2005;5:142-149.
58. Søreide K, Nedrebø BS, Knapp JC, et al. Evolving molecular classification by genomic and proteomic biomarkers in colorectal cancer: potential implications for the surgical oncologist. Surg Oncol. 2009;18:31-50.
59. Kupchak P, Wu AH, Ghani F, et al. Influence of imprecision on ROC curve analysis for cardiac markers. Clin Chem. 2006;52:752-753.
60. Kazmierczak SC, Catrou PG, Van Lente F. Enzymatic markers of gallstone-induced pancreatitis identified by ROC curve analysis, discriminant analysis, logistic regression, likelihood ratios, and information theory. Clin Chem. 1995;41:523-531.
61. Linden A. Measuring diagnostic and predictive accuracy in disease management: an introduction to receiver operating characteristic (ROC) analysis. J Eval Clin Pract. 2006;12:132-139.
62. Ridker PM, Cook NR. Biomarkers for prediction of cardiovascular events. N Engl J Med. 2007;356:1472-1473; author reply 4-5.
63. Walley T. Evaluating laboratory diagnostic tests. BMJ. 2008;336:569-570.
64. Pepe MS. Evaluating technologies for classification and prediction in medicine. Stat Med. 2005;24:3687-3696.
65. Ransohoff DF. Rules of evidence for cancer molecular-marker discovery and validation. Nat Rev Cancer. 2004;4:309-314.
66. Ware JH. The limitations of risk factors as prognostic tools. N Engl J Med. 2006;355:2615-2617.
67. Bossuyt PM, Reitsma JB, Bruns DE, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Standards for Reporting of Diagnostic Accuracy. Clin Chem. 2003;49:1-6.
68. McShane LM, Altman DG, Sauerbrei W, et al. Reporting recommendations for tumor marker prognostic studies. J Clin Oncol. 2005;23: 9067-9072.
69. Pepe MS, Etzioni R, Feng Z, et al. Phases of biomarker development for early detection of cancer. J Natl Cancer Inst. 2001;93:1054-1061.
70. Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem. 2008;54:17-23.
71. Janes H, Pepe MS. Adjusting for covariates in studies of diagnostic, screening, or prognostic markers: an old concept in a new setting. Am J Epidemiol. 2008;168:89-97.
72. Pepe MS, Zheng Y, Jin Y, et al. Evaluating the ROC performance of markers for future events. Lifetime Data Anal. 2008;14:86-113.
73. Rockette HE, Gur D. Selection of a rating scale in receiver operating characteristic studies: some remaining issues. Acad Radiol. 2008;15:245-248.
74. Hillis SL, Berbaum KS, Metz CE. Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Acad Radiol. 2008;15:647-661.
75. Pesce LL, Metz CE. Reliable and computationally efficient maximum-likelihood estimation of "proper" binormal ROC curves. Acad Radiol. 2007;14:814-829.
76. Obuchowski NA. New methodological tools for multiple-reader ROC studies. Radiology. 2007;243:10-12.
77. Kim J, Lee J, Chang E, et al. Selective sentinel node plus additional non-sentinel node biopsy based on an FDG-PET/CT scan in early breast cancer patients: single institutional experience. World J Surg. 2009;33:943-949.
78. Sitter H, Hoffmann S, Hassan I, et al. Diagnostic score in appendicitis. Validation of a diagnostic score (Eskelinen score) in patients in whom acute appendicitis is suspected. Langenbecks Arch Surg. 2004;389:213-218.
79. Goldman RD, Carter S, Stephens D, et al. Prospective validation of the pediatric appendicitis score. J Pediatr. 2008;153:278-282.
80. Glance LG, Osler TM, Mukamel DB, et al. TMPM-ICD9: a trauma mortality prediction model based on ICD-9-CM codes. Ann Surg. 2009;249:1032-1039.
81. Nunez TC, Voskresensky IV, Dossett LA, et al. Early prediction of massive transfusion in trauma: simple as ABC (assessment of blood consumption)? J Trauma. 2009;66:346-352.
82. Brenner DE, Normolle DP. Biomarkers for cancer risk, early detection, and prognosis: the validation conundrum. Cancer Epidemiol Biomarkers Prev. 2007;16:1918-1920.
83. Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999;282:1061-1066.
84. Rutjes AW, Reitsma JB, Di Nisio M, et al. Evidence of bias and variation in diagnostic accuracy studies. CMAJ. 2006;174:469-476.
85. Whiting P, Rutjes AW, Reitsma JB, et al. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med. 2004;140:189-202.
86. Pepe MS, Janes H. Insights into latent class analysis of diagnostic test performance. Biostatistics. 2007;8:474-484.
87. Cohn JN, Quyyumi AA, Hollenberg NK, et al. Surrogate markers for cardiovascular disease: functional markers. Circulation. 2004;109:IV31-IV46.
88. Mancini GB, Dahlof B, Diez J. Surrogate markers for cardiovascular disease: structural markers. Circulation. 2004;109:IV22-IV30.
89. Baker SG. Surrogate endpoints: wishful thinking or reality? J Natl Cancer Inst. 2006;98:502-503.
90. Schatzkin A. Problems with using biomarkers as surrogate end points for cancer: a cautionary tale. Recent Results Cancer Res. 2005;166:89-98.
91. Prentice RL. Surrogate and mediating endpoints: current status and future directions. J Natl Cancer Inst. 2009;101:216-217.
92. Kelloff GJ, Sigman CC, Johnson KM, et al. Perspectives on surrogate end points in the development of drugs that reduce the risk of cancer. Cancer Epidemiol Biomarkers Prev. 2000;9:127-137.
93. van den Broek MA, van Dam RM, Malago M, et al. Feasibility of randomized controlled trials in liver surgery using surgery-related mortality or morbidity as endpoint. Br J Surg. 2009;96:1005-1014.
94. Ludwig JA, Weinstein JN. Biomarkers in cancer staging, prognosis and treatment selection. Nat Rev Cancer. 2005;5:845-856.
95. Guller U. Caveats in the interpretation of the surgical literature. Br J Surg. 2008;95:541-546.
96. Mahid SS, Qadan M, Hornung CA, et al. Assessment of publication bias for the surgeon scientist. Br J Surg. 2008;95:943-949.
97. Leeflang MM, Deeks JJ, Gatsonis C, et al. Systematic reviews of diagnostic test accuracy. Ann Intern Med. 2008;149:889-897.
98. Zweig MH. ROC plots display test accuracy, but are still limited by the study design. Clin Chem. 1993;39:1345-1346.
99. Ison JC, Blades MJ. ROCPLOT: a generic software tool for ROC analysis and the validation of predictive methods. Appl Bioinformatics. 2005;4:131-135.
100. Stephan C, Wesseling S, Schink T, et al. Comparison of eight computer programs for receiver-operating characteristic analysis. Clin Chem. 2003;49:433-439.
101. Kramar A, Faraggi D, Fortune A, et al. mROC: a computer program for combining tumour markers in predicting disease states. Comput Methods Programs Biomed. 2001;66:199-207.
102. Swets JA, Dawes RM, Monahan J. Better decisions through science. Sci Am. 2000;283:82-87.