Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them)

Briefings in Bioinformatics

Volumn 13, Issue 1, 2012, Pages 83-97

  Berrar, Daniel ^a   Flach, Peter ^b  

^a Tokyo Institute of Technology ^*  (Japan)

^b UNIVERSITY OF BRISTOL   (United Kingdom)

Author keywords

Area under the curve; Microarrays; Model evaluation; Multiple testing; Receiver operating characteristic

Indexed keywords

AREA UNDER THE CURVE; ARTICLE; MEDICAL RESEARCH; METHODOLOGY; MICROARRAY ANALYSIS; RECEIVER OPERATING CHARACTERISTIC; REPRODUCIBILITY;

AREA UNDER CURVE; BIOMEDICAL RESEARCH; MICROARRAY ANALYSIS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; ROC CURVE;

EID: 84855679008     PISSN: 14675463     EISSN: 14774054     Source Type: Journal    
DOI: 10.1093/bib/bbr008     Document Type: Article

References (68)

1. Chang JC, Wooten EC, Tsimelzon A, et al. Gene expression profiling for the prediction of therapeutic response to docetaxel in patients with breast cancer. Lancet 2003;362(9381): 362-9.
2. Ayers M, Symmans WF, Stec J, et al. Gene expression profiles predict complete pathologic response to neoadjuvant paclitaxel and fluorouracil, doxorubicin, and cyclophosphamide chemotherapy in breast cancer. J Clin Oncol 2004; 22(12):2284-93.
3. Iwao-Koizumi K, Matoba R, Ueno N, et al. Prediction of docetaxel response in human breast cancer by gene expression profiling. J Clin Oncol 2005;23(3):422-31.
4. Jansen MP, Foekens JA, van Staveren IL, et al. Molecular classification of tamoxifen-resistant breast carcinomas by gene expression profiling. J Clin Oncol 2005;23(4):732-40.
5. van't Veer LJ, Dai H, van de Vijver MJ, et al. Gene expression profiling predicts clinical outcome of breast cancer. Nature 2002;415:530-6.
6. Bueno-de-Mesquita JM, van Harten WH, Retel VP, et al. Use of 70-gene signature to predict prognosis of patients with node-negative breast cancer: a prospective community-based feasibility study (RASTER). Lancet Oncol 2007;8(12):1079-87.
7. Chang HY, Nuyten DS, Sneddon JB, et al. Robustness, scalability, and integration of a wound-response gene expression signature in predicting breast cancer survival. Proc Natl Acad SciUSA 2005;102(10):3738-43.
8. Sørlie T, Perou CM, Tibshirani R, et al. Gene expression patterns of breast carcinomas distinguish tumor subclasses with clinical implications. Proc Natl Acad Sci USA 2001; 98(19):10869-74.
9. Sotiriou C, Wirapati P, Loi S, et al. Gene expression profiling in breast cancer: understanding the molecular basis of histologic grade to improve prognosis. J Natl Cancer Inst 2006;98(4):262-72.
10. Wang Y, Klijn JG, Zhang Y, et al. Gene-expression profiles to predict distant metastasis of lymph-node-negative primary breast cancer. Lancet 2005;365(9460):671-9.
11. Liu R, Wang X, Chen GY, et al. The prognostic role of a gene signature from tumorigenic breast-cancer cells. N Engl J Med 2007;356(3):217-26.
12. Chi JT, Wang Z, Nuyten DS, et al. Gene expression programs in response to hypoxia: cell type specificity and prognostic significance in human cancers. PLoSMed 2006;3(3): e47.
13. Michiels S, Koscielny S, Hill C. Prediction of cancer outcome with microarrays: a multiple random validation strategy. Lancet 2005;365(9458):488-92.
14. Ahmed AA, Brenton JD. Microarrays and breast cancer clinical studies: forgetting what we have not yet learnt. Br Cancer Res 2005;7(3):96-9.
15. Søreide K. Receiver-operating characteristic (ROC) curve analysis in diagnostic, prognostic and predictive biomarker research. J Clin Pathol 2009;62:1-5.
16. Wong AKS, Lee JWT, Yeung DS. Improving text classifier performance based on AUC. Proceedngs of Eighteenth International Conference on Pattern Classification, Hong Kong, 2006, 268-71.
17. Simon R. A roadmap for developing and validating therapeutically relevant genomic classifiers. J Clin Oncol 2005; 23(29):7332-41.
18. Flach P. ROC analysis. In: Sammut C, Webb GI, (eds). Encyclopedia of Machine Learning. Berlin/Heidelberg: Springer, 2010;869-74.
19. Fawcett T. ROC Graphs: Notes and practical considerations for researchers. Technical Report HPL-2003-4. HP Laboratories, 2004.
20. Ling CX, Huang J, Zhang H. AUC: A statistically consistent and more discriminating measure than accuracy. Proc 18th Intl Conf Art Int 2003;18:519-26.
21. Obuchowski NA, Lieber ML, Wians FH, Jr. ROC curves in clinical chemistry: Uses, misuses, and possible solutions. Clin Chem 2004;50:1118-25.
22. Recode for calculating H-measure. http://www2.imperial.ac.uk/-djhand/. (31 January 2011, date last accessed).
23. Adams NM, Hand DJ. (1999) Comparing classifiers when the misallocation costs are uncertain. Pattern Recog 1999; 32(7):1139-47.
24. Drummond C, Holte RC. Cost curves: An improved method for visualizing classifier performance. Mach Learn 2006;65:95-130.
25. Lobo JM, Jiménez-Valverde A, Real R. AUC: a misleading measure of the performance of predictive distribution models. Global Ecol Biogeogr 2008;17:145-51.
26. Hilden J, Glasziou P. Regret graphs, diagnostic uncertainty and Youden's index. StatMed 1996;15:969-86.
27. McClish DK. Analyzing a portion of the ROC curve. Med DecisMaking 1989;9(3):190-5.
28. Vanderlooy S, Hüllermeier E. A critical analysis of variants of AUC. Mach Learn 2008;72:247-62.
29. R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria, 2009. http://www.R-project .org.
30. Ransohoff DF. Bias as a threat to the validity of cancer molecular-marker research. Nat Rev Cancer 2005;5(2): 142-9.
31. Baggerly KA, Coombes KR. Deriving chemosensitivity from cell lines: forensic bioinformatics and reproducible research in high-throughput biology. Ann Appl Stat 2009; 3(4):1309-34.
32. Potti A, Dressman HK, Bild A, et al. Genomic signatures to guide the use of chemotherapeutics. Nat Med 2006;12(11): 1294-300.
33. Hsu DS, Balakumaran BS, Acharya CR, et al. Pharmacogenomic strategies provide a rational approach to the treatment of cisplatin-resistant patients with advanced cancer. JClin Oncol 2007;25(28):4350-7.
34. Dressman HK, Berchuck A, Chan G, et al. An integrated genomic-based approach to individualized treatment of patients with advanced-stage ovarian cancer. J Clin Oncol 2007;25(5):517-25.
35. Baggerly KA, Coombes KR, Neeley ES. Run batch effects potentially compromise the usefulness of genomic signatures for ovarian cancer. JClin Oncol 2008;25:1186-7.
36. Bonnefoi H, Potti A, Delorenzi M, et al. Validation of gene signatures that predict the response of breast cancer to neoadjuvant chemotherapy: a substudy of the EORTC 10994/BIG 00-01 clinical trial. Lancet Oncol 2007;8(12): 1071-8.
37. Clark RD, Webster-Clark DJ. Managing bias in ROC curves. J Comput Aided Mol Des 2008;22:141-6.
38. Søreide K, Korner H, Søreide JA. Diagnostic accuracy and receiver-operating characteristics curve analysis in surgical research and decision making. Ann Surg 2011;253(1): 27-34.
39. Hess KR, Anderson K, Symmans WF, et al. Pharmacogenomic predictor of sensitivity to preoperative chemotherapy with paclitaxel and fluorouracil, doxorubicin, and cyclophosphamide in breast cancer. J Clin Oncol 2006;24(26):4236-44.
40. Swamidass SJ, Azencott CA, Daily K, Baldi P. A CROC stronger than ROC: measuring, visualizing and optimizing early retrieval. Bioinformatics 2010;26(10):1348-56.
41. Holm S. A simple sequentially rejective multiple test procedure. Scand J Stat 1979;6:65-70.
42. Manly KF, Nettleton D, Hwang JT. Genomics, prior probability, and statistical tests of multiple hypotheses. Genome Res 2004;14(6):997-1001.
43. George SL. Statistical issues in translational cancer research. ClinCancer Res 2008;14(19):5954-8.
44. Holland B. On the application of three modified Bonferroni procedures to pairwise multiple comparisons in balanced repeated measures designs. Comp Stat Quarterly 1991;6: 219-31.
45. Rothman J. Writing for Epidemiology. Epidemiology 1998; 9(3):333-7.
46. Rothman J. A show of confidence. N Engl J Med 1978;299: 1362-3.
47. Gardner MJ, Altman DG. Confidence intervals rather than P values: estimation rather than hypothesis testing. Brit Med J 1986;292:746-50.
48. Poole C. Low p-values or narrow confidence intervals: which are more durable? Epidemiology 2001;12(3): 291-4.
49. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1984;143(1):29-36.
50. Cortes C, Mohri M. Confidence intervals for the area under the ROC curve. Adv Neural Inf Process Syst 2005;17:305-12.
51. Efron B. Better bootstrap confidence intervals. J Am Stat Assoc 1987;82:171-200.
52. Salter KH, Acharya CR, Walters KS, et al. An integrated approach to the prediction of chemotherapeutic response in patients with breast cancer. PLoSONE 2008;3(4):e1908.
53. Karlsson E, Delle U, Danielsson A, et al. Gene expression variation to predict 10-year survival in lymph-nodenegative breast cancer. BMCCancer 2008;8:254.
54. Garman KS, Acharya CR, Edelman E, et al. A genomic approach to colon cancer risk stratification yields biologic insights into therapeutic opportunities. Proc Natl Acad Sci USA 2008;105(49):19432-7.
55. Hand DJ. Classifier technology and the illusion of progress. Stat Sci 2006;21(1):1-15.
56. R Bioconductor project. http://www.bioconductor.org (31 January 2011, date last accessed).
57. Sing T, Sander O, Beerenwinkel N, Lengauer T. ROCR: visualizing classifier performance in R. Bioinformatics 2005; 21(20):3940-1.
58. Tuszynski J. colAUC {caTools}, R library. http://cran.rproject. org/web/packages/caTools/(6 March 2011, date last accessed).
59. Simpson G. plot.roc {analogue}, R library. http://cran.rproject. org/web/packages/analogue/(6 March 2011, date last accessed).
60. Hall M, Frank E, Holmes G, et al. The WEKA data mining software: an update. SIGKDDExpl 2009;11(1):10-8.
61. Vergara IA, Norambuena T, Ferrada E, et al. StAR: a simple tool for the statistical comparison of ROC curves. BMC Bioinformatics 2008;9:265.
62. GraphPad Prism. http://www.graphpad.com/prism/ (6 March 2011, date last accessed).
63. Hilden J. The area under the ROC curve and its competitors. Med DecisMaking 1991;11(2):95-101.
64. Hand DJ. Measuring classifier performance: a coherent alternative to the area under the ROC curve. Mach Learn 2009;77:103-23.
65. Simon R. Supervised analysis when the number of candidate features (p) greatly exceeds the number of cases (n). ACM SIGKDD Expl Newsletter 2003;5(2):31-7.
66. Hanczar B, Hua J, Sima C, et al. Small-sample precision of ROC-related estimates. Bioinformatics 2010;26(6):822-30.
67. Hilden J. What properties should an overall measure of test performance possess? Clin Chem 2005;51(2):471.
68. Flach P. The many faces of ROC analysis in machine learning. TutorialNotes of the 21st International Conference onMachine Learning, 2004 (ICML04) 2004. http://www.cs.bris.ac.uk/-flach/ICML04tutorial/index.html (31 January 2011, date last accessed).