메뉴 건너뛰기




Volumn 13, Issue 1, 2012, Pages 83-97

Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them)

Author keywords

Area under the curve; Microarrays; Model evaluation; Multiple testing; Receiver operating characteristic

Indexed keywords

AREA UNDER THE CURVE; ARTICLE; MEDICAL RESEARCH; METHODOLOGY; MICROARRAY ANALYSIS; RECEIVER OPERATING CHARACTERISTIC; REPRODUCIBILITY;

EID: 84855679008     PISSN: 14675463     EISSN: 14774054     Source Type: Journal    
DOI: 10.1093/bib/bbr008     Document Type: Article
Times cited : (96)

References (68)
  • 1
    • 0042125511 scopus 로고    scopus 로고
    • Gene expression profiling for the prediction of therapeutic response to docetaxel in patients with breast cancer
    • Chang JC, Wooten EC, Tsimelzon A, et al. Gene expression profiling for the prediction of therapeutic response to docetaxel in patients with breast cancer. Lancet 2003;362(9381): 362-9.
    • (2003) Lancet , vol.362 , Issue.9381 , pp. 362-369
    • Chang, J.C.1    Wooten, E.C.2    Tsimelzon, A.3
  • 2
    • 2942729848 scopus 로고    scopus 로고
    • Gene expression profiles predict complete pathologic response to neoadjuvant paclitaxel and fluorouracil, doxorubicin, and cyclophosphamide chemotherapy in breast cancer
    • Ayers M, Symmans WF, Stec J, et al. Gene expression profiles predict complete pathologic response to neoadjuvant paclitaxel and fluorouracil, doxorubicin, and cyclophosphamide chemotherapy in breast cancer. J Clin Oncol 2004; 22(12):2284-93.
    • (2004) J Clin Oncol , vol.22 , Issue.12 , pp. 2284-2293
    • Ayers, M.1    Symmans, W.F.2    Stec, J.3
  • 3
    • 14544268597 scopus 로고    scopus 로고
    • Prediction of docetaxel response in human breast cancer by gene expression profiling
    • Iwao-Koizumi K, Matoba R, Ueno N, et al. Prediction of docetaxel response in human breast cancer by gene expression profiling. J Clin Oncol 2005;23(3):422-31.
    • (2005) J Clin Oncol , vol.23 , Issue.3 , pp. 422-431
    • Iwao-Koizumi, K.1    Matoba, R.2    Ueno, N.3
  • 4
    • 20044377912 scopus 로고    scopus 로고
    • Molecular classification of tamoxifen-resistant breast carcinomas by gene expression profiling
    • Jansen MP, Foekens JA, van Staveren IL, et al. Molecular classification of tamoxifen-resistant breast carcinomas by gene expression profiling. J Clin Oncol 2005;23(4):732-40.
    • (2005) J Clin Oncol , vol.23 , Issue.4 , pp. 732-740
    • Jansen, M.P.1    Foekens, J.A.2    van Staveren, I.L.3
  • 5
    • 18244409687 scopus 로고    scopus 로고
    • Gene expression profiling predicts clinical outcome of breast cancer
    • van't Veer LJ, Dai H, van de Vijver MJ, et al. Gene expression profiling predicts clinical outcome of breast cancer. Nature 2002;415:530-6.
    • (2002) Nature , vol.415 , pp. 530-536
    • van't Veer, L.J.1    Dai, H.2    van de Vijver, M.J.3
  • 6
    • 36549030796 scopus 로고    scopus 로고
    • Use of 70-gene signature to predict prognosis of patients with node-negative breast cancer: a prospective community-based feasibility study (RASTER)
    • Bueno-de-Mesquita JM, van Harten WH, Retel VP, et al. Use of 70-gene signature to predict prognosis of patients with node-negative breast cancer: a prospective community-based feasibility study (RASTER). Lancet Oncol 2007;8(12):1079-87.
    • (2007) Lancet Oncol , vol.8 , Issue.12 , pp. 1079-1087
    • Bueno-de-Mesquita, J.M.1    van Harten, W.H.2    Retel, V.P.3
  • 7
    • 20144386127 scopus 로고    scopus 로고
    • Robustness, scalability, and integration of a wound-response gene expression signature in predicting breast cancer survival
    • Chang HY, Nuyten DS, Sneddon JB, et al. Robustness, scalability, and integration of a wound-response gene expression signature in predicting breast cancer survival. Proc Natl Acad SciUSA 2005;102(10):3738-43.
    • (2005) Proc Natl Acad Sci USA , vol.102 , Issue.10 , pp. 3738-3743
    • Chang, H.Y.1    Nuyten, D.S.2    Sneddon, J.B.3
  • 8
    • 0035845511 scopus 로고    scopus 로고
    • Gene expression patterns of breast carcinomas distinguish tumor subclasses with clinical implications
    • Sørlie T, Perou CM, Tibshirani R, et al. Gene expression patterns of breast carcinomas distinguish tumor subclasses with clinical implications. Proc Natl Acad Sci USA 2001; 98(19):10869-74.
    • (2001) Proc Natl Acad Sci USA , vol.98 , Issue.19 , pp. 10869-10874
    • Sørlie, T.1    Perou, C.M.2    Tibshirani, R.3
  • 9
    • 33144462268 scopus 로고    scopus 로고
    • Gene expression profiling in breast cancer: understanding the molecular basis of histologic grade to improve prognosis
    • Sotiriou C, Wirapati P, Loi S, et al. Gene expression profiling in breast cancer: understanding the molecular basis of histologic grade to improve prognosis. J Natl Cancer Inst 2006;98(4):262-72.
    • (2006) J Natl Cancer Inst , vol.98 , Issue.4 , pp. 262-272
    • Sotiriou, C.1    Wirapati, P.2    Loi, S.3
  • 10
    • 13844310310 scopus 로고    scopus 로고
    • Gene-expression profiles to predict distant metastasis of lymph-node-negative primary breast cancer
    • Wang Y, Klijn JG, Zhang Y, et al. Gene-expression profiles to predict distant metastasis of lymph-node-negative primary breast cancer. Lancet 2005;365(9460):671-9.
    • (2005) Lancet , vol.365 , Issue.9460 , pp. 671-679
    • Wang, Y.1    Klijn, J.G.2    Zhang, Y.3
  • 11
    • 33846425793 scopus 로고    scopus 로고
    • The prognostic role of a gene signature from tumorigenic breast-cancer cells
    • Liu R, Wang X, Chen GY, et al. The prognostic role of a gene signature from tumorigenic breast-cancer cells. N Engl J Med 2007;356(3):217-26.
    • (2007) N Engl J Med , vol.356 , Issue.3 , pp. 217-226
    • Liu, R.1    Wang, X.2    Chen, G.Y.3
  • 12
    • 33644551073 scopus 로고    scopus 로고
    • Gene expression programs in response to hypoxia: cell type specificity and prognostic significance in human cancers
    • Chi JT, Wang Z, Nuyten DS, et al. Gene expression programs in response to hypoxia: cell type specificity and prognostic significance in human cancers. PLoSMed 2006;3(3): e47.
    • (2006) PLoSMed , vol.3 , Issue.3
    • Chi, J.T.1    Wang, Z.2    Nuyten, D.S.3
  • 13
    • 13444249852 scopus 로고    scopus 로고
    • Prediction of cancer outcome with microarrays: a multiple random validation strategy
    • Michiels S, Koscielny S, Hill C. Prediction of cancer outcome with microarrays: a multiple random validation strategy. Lancet 2005;365(9458):488-92.
    • (2005) Lancet , vol.365 , Issue.9458 , pp. 488-492
    • Michiels, S.1    Koscielny, S.2    Hill, C.3
  • 14
    • 18944391930 scopus 로고    scopus 로고
    • Microarrays and breast cancer clinical studies: forgetting what we have not yet learnt
    • Ahmed AA, Brenton JD. Microarrays and breast cancer clinical studies: forgetting what we have not yet learnt. Br Cancer Res 2005;7(3):96-9.
    • (2005) Br Cancer Res , vol.7 , Issue.3 , pp. 96-99
    • Ahmed, A.A.1    Brenton, J.D.2
  • 15
    • 58349085083 scopus 로고    scopus 로고
    • Receiver-operating characteristic (ROC) curve analysis in diagnostic, prognostic and predictive biomarker research
    • Søreide K. Receiver-operating characteristic (ROC) curve analysis in diagnostic, prognostic and predictive biomarker research. J Clin Pathol 2009;62:1-5.
    • (2009) J Clin Pathol , vol.62 , pp. 1-5
    • Søreide, K.1
  • 17
    • 33644688970 scopus 로고    scopus 로고
    • A roadmap for developing and validating therapeutically relevant genomic classifiers
    • Simon R. A roadmap for developing and validating therapeutically relevant genomic classifiers. J Clin Oncol 2005; 23(29):7332-41.
    • (2005) J Clin Oncol , vol.23 , Issue.29 , pp. 7332-7341
    • Simon, R.1
  • 18
    • 83055177185 scopus 로고    scopus 로고
    • ROC analysis
    • Sammut C, Webb GI, (eds). Berlin/Heidelberg: Springer
    • Flach P. ROC analysis. In: Sammut C, Webb GI, (eds). Encyclopedia of Machine Learning. Berlin/Heidelberg: Springer, 2010;869-74.
    • (2010) Encyclopedia of Machine Learning , pp. 869-874
    • Flach, P.1
  • 20
    • 84880794162 scopus 로고    scopus 로고
    • AUC: A statistically consistent and more discriminating measure than accuracy
    • Ling CX, Huang J, Zhang H. AUC: A statistically consistent and more discriminating measure than accuracy. Proc 18th Intl Conf Art Int 2003;18:519-26.
    • (2003) Proc 18th Intl Conf Art Int , vol.18 , pp. 519-526
    • Ling, C.X.1    Huang, J.2    Zhang, H.3
  • 21
    • 3042512345 scopus 로고    scopus 로고
    • ROC curves in clinical chemistry: Uses, misuses, and possible solutions
    • Obuchowski NA, Lieber ML, Wians FH, Jr. ROC curves in clinical chemistry: Uses, misuses, and possible solutions. Clin Chem 2004;50:1118-25.
    • (2004) Clin Chem , vol.50 , pp. 1118-1125
    • Obuchowski, N.A.1    Lieber, M.L.2    Wians Jr., F.H.3
  • 22
    • 84878674718 scopus 로고    scopus 로고
    • Recode for calculating H-measure. (31 January date last accessed)
    • Recode for calculating H-measure. http://www2.imperial.ac.uk/-djhand/. (31 January 2011, date last accessed).
    • (2011)
  • 23
    • 0033164667 scopus 로고    scopus 로고
    • Comparing classifiers when the misallocation costs are uncertain
    • Adams NM, Hand DJ. (1999) Comparing classifiers when the misallocation costs are uncertain. Pattern Recog 1999; 32(7):1139-47.
    • (1999) Pattern Recog 1999 , vol.32 , Issue.7 , pp. 1139-1147
    • Adams, N.M.1    Hand, D.J.2
  • 24
    • 33748991193 scopus 로고    scopus 로고
    • Cost curves: An improved method for visualizing classifier performance
    • Drummond C, Holte RC. Cost curves: An improved method for visualizing classifier performance. Mach Learn 2006;65:95-130.
    • (2006) Mach Learn , vol.65 , pp. 95-130
    • Drummond, C.1    Holte, R.C.2
  • 25
    • 38949161848 scopus 로고    scopus 로고
    • AUC: a misleading measure of the performance of predictive distribution models
    • Lobo JM, Jiménez-Valverde A, Real R. AUC: a misleading measure of the performance of predictive distribution models. Global Ecol Biogeogr 2008;17:145-51.
    • (2008) Global Ecol Biogeogr , vol.17 , pp. 145-151
    • Lobo, J.M.1    Jiménez-Valverde, A.2    Real, R.3
  • 26
    • 0029890305 scopus 로고    scopus 로고
    • Regret graphs, diagnostic uncertainty and Youden's index
    • Hilden J, Glasziou P. Regret graphs, diagnostic uncertainty and Youden's index. StatMed 1996;15:969-86.
    • (1996) StatMed , vol.15 , pp. 969-986
    • Hilden, J.1    Glasziou, P.2
  • 27
    • 0024370623 scopus 로고
    • Analyzing a portion of the ROC curve
    • McClish DK. Analyzing a portion of the ROC curve. Med DecisMaking 1989;9(3):190-5.
    • (1989) Med DecisMaking , vol.9 , Issue.3 , pp. 190-195
    • McClish, D.K.1
  • 28
    • 48349140000 scopus 로고    scopus 로고
    • A critical analysis of variants of AUC
    • Vanderlooy S, Hüllermeier E. A critical analysis of variants of AUC. Mach Learn 2008;72:247-62.
    • (2008) Mach Learn , vol.72 , pp. 247-262
    • Vanderlooy, S.1    Hüllermeier, E.2
  • 29
    • 70149113077 scopus 로고    scopus 로고
    • R Development Core TeamR Foundation for Statistical Computing, Vienna, Austria
    • R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria, 2009. http://www.R-project .org.
    • (2009) R: A language and environment for statistical computing
  • 30
    • 13444283596 scopus 로고    scopus 로고
    • Bias as a threat to the validity of cancer molecular-marker research
    • Ransohoff DF. Bias as a threat to the validity of cancer molecular-marker research. Nat Rev Cancer 2005;5(2): 142-9.
    • (2005) Nat Rev Cancer , vol.5 , Issue.2 , pp. 142-149
    • Ransohoff, D.F.1
  • 31
    • 77951163222 scopus 로고    scopus 로고
    • Deriving chemosensitivity from cell lines: forensic bioinformatics and reproducible research in high-throughput biology
    • Baggerly KA, Coombes KR. Deriving chemosensitivity from cell lines: forensic bioinformatics and reproducible research in high-throughput biology. Ann Appl Stat 2009; 3(4):1309-34.
    • (2009) Ann Appl Stat , vol.3 , Issue.4 , pp. 1309-1334
    • Baggerly, K.A.1    Coombes, K.R.2
  • 32
    • 33750727368 scopus 로고    scopus 로고
    • Genomic signatures to guide the use of chemotherapeutics
    • Potti A, Dressman HK, Bild A, et al. Genomic signatures to guide the use of chemotherapeutics. Nat Med 2006;12(11): 1294-300.
    • (2006) Nat Med , vol.12 , Issue.11 , pp. 1294-1300
    • Potti, A.1    Dressman, H.K.2    Bild, A.3
  • 33
    • 35348860753 scopus 로고    scopus 로고
    • Pharmacogenomic strategies provide a rational approach to the treatment of cisplatin-resistant patients with advanced cancer
    • Hsu DS, Balakumaran BS, Acharya CR, et al. Pharmacogenomic strategies provide a rational approach to the treatment of cisplatin-resistant patients with advanced cancer. JClin Oncol 2007;25(28):4350-7.
    • (2007) JClin Oncol , vol.25 , Issue.28 , pp. 4350-4357
    • Hsu, D.S.1    Balakumaran, B.S.2    Acharya, C.R.3
  • 34
    • 33947512381 scopus 로고    scopus 로고
    • An integrated genomic-based approach to individualized treatment of patients with advanced-stage ovarian cancer
    • Dressman HK, Berchuck A, Chan G, et al. An integrated genomic-based approach to individualized treatment of patients with advanced-stage ovarian cancer. J Clin Oncol 2007;25(5):517-25.
    • (2007) J Clin Oncol , vol.25 , Issue.5 , pp. 517-525
    • Dressman, H.K.1    Berchuck, A.2    Chan, G.3
  • 35
    • 41949101394 scopus 로고    scopus 로고
    • Run batch effects potentially compromise the usefulness of genomic signatures for ovarian cancer
    • Baggerly KA, Coombes KR, Neeley ES. Run batch effects potentially compromise the usefulness of genomic signatures for ovarian cancer. JClin Oncol 2008;25:1186-7.
    • (2008) JClin Oncol , vol.25 , pp. 1186-1187
    • Baggerly, K.A.1    Coombes, K.R.2    Neeley, E.S.3
  • 36
    • 36549068221 scopus 로고    scopus 로고
    • Validation of gene signatures that predict the response of breast cancer to neoadjuvant chemotherapy: a substudy of the EORTC 10994/BIG 00-01 clinical trial
    • Bonnefoi H, Potti A, Delorenzi M, et al. Validation of gene signatures that predict the response of breast cancer to neoadjuvant chemotherapy: a substudy of the EORTC 10994/BIG 00-01 clinical trial. Lancet Oncol 2007;8(12): 1071-8.
    • (2007) Lancet Oncol , vol.8 , Issue.12 , pp. 1071-1078
    • Bonnefoi, H.1    Potti, A.2    Delorenzi, M.3
  • 38
    • 78751630983 scopus 로고    scopus 로고
    • Diagnostic accuracy and receiver-operating characteristics curve analysis in surgical research and decision making
    • Søreide K, Korner H, Søreide JA. Diagnostic accuracy and receiver-operating characteristics curve analysis in surgical research and decision making. Ann Surg 2011;253(1): 27-34.
    • (2011) Ann Surg , vol.253 , Issue.1 , pp. 27-34
    • Søreide, K.1    Korner, H.2    Søreide, J.A.3
  • 39
    • 33749030177 scopus 로고    scopus 로고
    • Pharmacogenomic predictor of sensitivity to preoperative chemotherapy with paclitaxel and fluorouracil, doxorubicin, and cyclophosphamide in breast cancer
    • Hess KR, Anderson K, Symmans WF, et al. Pharmacogenomic predictor of sensitivity to preoperative chemotherapy with paclitaxel and fluorouracil, doxorubicin, and cyclophosphamide in breast cancer. J Clin Oncol 2006;24(26):4236-44.
    • (2006) J Clin Oncol , vol.24 , Issue.26 , pp. 4236-4244
    • Hess, K.R.1    Anderson, K.2    Symmans, W.F.3
  • 40
    • 77952832818 scopus 로고    scopus 로고
    • A CROC stronger than ROC: measuring, visualizing and optimizing early retrieval
    • Swamidass SJ, Azencott CA, Daily K, Baldi P. A CROC stronger than ROC: measuring, visualizing and optimizing early retrieval. Bioinformatics 2010;26(10):1348-56.
    • (2010) Bioinformatics , vol.26 , Issue.10 , pp. 1348-1356
    • Swamidass, S.J.1    Azencott, C.A.2    Daily, K.3    Baldi, P.4
  • 41
    • 0002294347 scopus 로고
    • A simple sequentially rejective multiple test procedure
    • Holm S. A simple sequentially rejective multiple test procedure. Scand J Stat 1979;6:65-70.
    • (1979) Scand J Stat , vol.6 , pp. 65-70
    • Holm, S.1
  • 42
    • 3042562270 scopus 로고    scopus 로고
    • Genomics, prior probability, and statistical tests of multiple hypotheses
    • Manly KF, Nettleton D, Hwang JT. Genomics, prior probability, and statistical tests of multiple hypotheses. Genome Res 2004;14(6):997-1001.
    • (2004) Genome Res , vol.14 , Issue.6 , pp. 997-1001
    • Manly, K.F.1    Nettleton, D.2    Hwang, J.T.3
  • 43
    • 58149163344 scopus 로고    scopus 로고
    • Statistical issues in translational cancer research
    • George SL. Statistical issues in translational cancer research. ClinCancer Res 2008;14(19):5954-8.
    • (2008) ClinCancer Res , vol.14 , Issue.19 , pp. 5954-5958
    • George, S.L.1
  • 44
    • 18244400004 scopus 로고
    • On the application of three modified Bonferroni procedures to pairwise multiple comparisons in balanced repeated measures designs
    • Holland B. On the application of three modified Bonferroni procedures to pairwise multiple comparisons in balanced repeated measures designs. Comp Stat Quarterly 1991;6: 219-31.
    • (1991) Comp Stat Quarterly , vol.6 , pp. 219-231
    • Holland, B.1
  • 45
    • 0031924982 scopus 로고    scopus 로고
    • Writing for Epidemiology
    • Rothman J. Writing for Epidemiology. Epidemiology 1998; 9(3):333-7.
    • (1998) Epidemiology , vol.9 , Issue.3 , pp. 333-337
    • Rothman, J.1
  • 46
    • 0018267981 scopus 로고
    • A show of confidence
    • Rothman J. A show of confidence. N Engl J Med 1978;299: 1362-3.
    • (1978) N Engl J Med , vol.299 , pp. 1362-1363
    • Rothman, J.1
  • 47
    • 0023049334 scopus 로고
    • Confidence intervals rather than P values: estimation rather than hypothesis testing
    • Gardner MJ, Altman DG. Confidence intervals rather than P values: estimation rather than hypothesis testing. Brit Med J 1986;292:746-50.
    • (1986) Brit Med J , vol.292 , pp. 746-750
    • Gardner, M.J.1    Altman, D.G.2
  • 48
    • 0035063893 scopus 로고    scopus 로고
    • Low p-values or narrow confidence intervals: which are more durable?
    • Poole C. Low p-values or narrow confidence intervals: which are more durable? Epidemiology 2001;12(3): 291-4.
    • (2001) Epidemiology , vol.12 , Issue.3 , pp. 291-294
    • Poole, C.1
  • 49
    • 0020083498 scopus 로고
    • The meaning and use of the area under a receiver operating characteristic (ROC) curve
    • Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1984;143(1):29-36.
    • (1984) Radiology , vol.143 , Issue.1 , pp. 29-36
    • Hanley, J.A.1    McNeil, B.J.2
  • 50
    • 84898972495 scopus 로고    scopus 로고
    • Confidence intervals for the area under the ROC curve
    • Cortes C, Mohri M. Confidence intervals for the area under the ROC curve. Adv Neural Inf Process Syst 2005;17:305-12.
    • (2005) Adv Neural Inf Process Syst , vol.17 , pp. 305-312
    • Cortes, C.1    Mohri, M.2
  • 51
    • 84923818429 scopus 로고
    • Better bootstrap confidence intervals
    • Efron B. Better bootstrap confidence intervals. J Am Stat Assoc 1987;82:171-200.
    • (1987) J Am Stat Assoc , vol.82 , pp. 171-200
    • Efron, B.1
  • 52
    • 44849135294 scopus 로고    scopus 로고
    • An integrated approach to the prediction of chemotherapeutic response in patients with breast cancer
    • Salter KH, Acharya CR, Walters KS, et al. An integrated approach to the prediction of chemotherapeutic response in patients with breast cancer. PLoSONE 2008;3(4):e1908.
    • (2008) PLoSONE , vol.3 , Issue.4
    • Salter, K.H.1    Acharya, C.R.2    Walters, K.S.3
  • 53
    • 53549094961 scopus 로고    scopus 로고
    • Gene expression variation to predict 10-year survival in lymph-nodenegative breast cancer
    • Karlsson E, Delle U, Danielsson A, et al. Gene expression variation to predict 10-year survival in lymph-nodenegative breast cancer. BMCCancer 2008;8:254.
    • (2008) BMCCancer , vol.8 , pp. 254
    • Karlsson, E.1    Delle, U.2    Danielsson, A.3
  • 54
    • 58049221094 scopus 로고    scopus 로고
    • A genomic approach to colon cancer risk stratification yields biologic insights into therapeutic opportunities
    • Garman KS, Acharya CR, Edelman E, et al. A genomic approach to colon cancer risk stratification yields biologic insights into therapeutic opportunities. Proc Natl Acad Sci USA 2008;105(49):19432-7.
    • (2008) Proc Natl Acad Sci USA , vol.105 , Issue.49 , pp. 19432-19437
    • Garman, K.S.1    Acharya, C.R.2    Edelman, E.3
  • 55
    • 33745886270 scopus 로고    scopus 로고
    • Classifier technology and the illusion of progress
    • Hand DJ. Classifier technology and the illusion of progress. Stat Sci 2006;21(1):1-15.
    • (2006) Stat Sci , vol.21 , Issue.1 , pp. 1-15
    • Hand, D.J.1
  • 56
    • 84878670213 scopus 로고    scopus 로고
    • R Bioconductor, project. (31 January date last accessed)
    • R Bioconductor project. http://www.bioconductor.org (31 January 2011, date last accessed).
    • (2011)
  • 59
    • 84878672828 scopus 로고    scopus 로고
    • 6 March, date last accessed
    • Simpson G. plot.roc {analogue}, R library. http://cran.rproject. org/web/packages/analogue/(6 March 2011, date last accessed).
    • (2011) plot.roc {analogue}, R library
    • Simpson, G.1
  • 60
    • 76749092270 scopus 로고    scopus 로고
    • The WEKA data mining software: an update
    • Hall M, Frank E, Holmes G, et al. The WEKA data mining software: an update. SIGKDDExpl 2009;11(1):10-8.
    • (2009) SIGKDDExpl , vol.11 , Issue.1 , pp. 10-18
    • Hall, M.1    Frank, E.2    Holmes, G.3
  • 61
    • 46049119818 scopus 로고    scopus 로고
    • StAR: a simple tool for the statistical comparison of ROC curves
    • Vergara IA, Norambuena T, Ferrada E, et al. StAR: a simple tool for the statistical comparison of ROC curves. BMC Bioinformatics 2008;9:265.
    • (2008) BMC Bioinformatics , vol.9 , pp. 265
    • Vergara, I.A.1    Norambuena, T.2    Ferrada, E.3
  • 62
    • 84878670618 scopus 로고    scopus 로고
    • (6 March date last accessed)
    • GraphPad Prism. http://www.graphpad.com/prism/ (6 March 2011, date last accessed).
    • (2011)
    • GraphPad, P.1
  • 63
    • 0025756234 scopus 로고
    • The area under the ROC curve and its competitors
    • Hilden J. The area under the ROC curve and its competitors. Med DecisMaking 1991;11(2):95-101.
    • (1991) Med DecisMaking , vol.11 , Issue.2 , pp. 95-101
    • Hilden, J.1
  • 64
    • 69549133517 scopus 로고    scopus 로고
    • Measuring classifier performance: a coherent alternative to the area under the ROC curve
    • Hand DJ. Measuring classifier performance: a coherent alternative to the area under the ROC curve. Mach Learn 2009;77:103-23.
    • (2009) Mach Learn , vol.77 , pp. 103-123
    • Hand, D.J.1
  • 65
    • 5344247436 scopus 로고    scopus 로고
    • Supervised analysis when the number of candidate features (p) greatly exceeds the number of cases (n)
    • Simon R. Supervised analysis when the number of candidate features (p) greatly exceeds the number of cases (n). ACM SIGKDD Expl Newsletter 2003;5(2):31-7.
    • (2003) ACM SIGKDD Expl News letter , vol.5 , Issue.2 , pp. 31-37
    • Simon, R.1
  • 66
    • 77951964158 scopus 로고    scopus 로고
    • Small-sample precision of ROC-related estimates
    • Hanczar B, Hua J, Sima C, et al. Small-sample precision of ROC-related estimates. Bioinformatics 2010;26(6):822-30.
    • (2010) Bioinformatics , vol.26 , Issue.6 , pp. 822-830
    • Hanczar, B.1    Hua, J.2    Sima, C.3
  • 67
    • 12944316442 scopus 로고    scopus 로고
    • What properties should an overall measure of test performance possess?
    • Hilden J. What properties should an overall measure of test performance possess? Clin Chem 2005;51(2):471.
    • (2005) Clin Chem , vol.51 , Issue.2 , pp. 471
    • Hilden, J.1
  • 68
    • 84878674245 scopus 로고    scopus 로고
    • TutorialNotes of the 21st International Conference onMachine Learning, 2004 (ICML04) 2004 (31 January date last accessed)
    • Flach P. The many faces of ROC analysis in machine learning. TutorialNotes of the 21st International Conference onMachine Learning, 2004 (ICML04) 2004. http://www.cs.bris.ac.uk/-flach/ICML04tutorial/index.html (31 January 2011, date last accessed).
    • (2011) The many faces of ROC analysis in machine learning
    • Flach, P.1


* 이 정보는 Elsevier사의 SCOPUS DB에서 KISTI가 분석하여 추출한 것입니다.