Optimal classifier selection and negative bias in error rate estimation: An empirical study on high-dimensional prediction

BMC Medical Research Methodology

Volumn 9, Issue , 2009, Pages

  Boulesteix, Anne Laure ^a,b   Strobl, Carolin ^a  

^a UNIVERSITY OF MUNICH   (Germany)

^b Sylvia Lawry Centre for Multiple Sclerosis Research   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; CLASSIFICATION; COLON TUMOR; EMPIRICAL RESEARCH; EPIDEMIOLOGY; GENETIC SCREENING; GENETICS; HUMAN; MALE; METHODOLOGY; MICROARRAY ANALYSIS; PREDICTION AND FORECASTING; PROSTATE TUMOR; STATISTICAL MODEL; STATISTICS; TUMOR GENE;

BIAS (EPIDEMIOLOGY); COLONIC NEOPLASMS; EMPIRICAL RESEARCH; GENES, NEOPLASM; GENETIC TESTING; HUMANS; MALE; MICROARRAY ANALYSIS; MODELS, STATISTICAL; PREDICTIVE VALUE OF TESTS; PROSTATIC NEOPLASMS;

EID: 76649101119     PISSN: None     EISSN: 14712288     Source Type: Journal    
DOI: 10.1186/1471-2288-9-85     Document Type: Article

References (39)

1. Almost all articles on cancer prognostic markers report statistically significant results. PA Kyzas D Denaxa-Kyza JP Ioannidis, European Journal of Cancer 2007 43 2559 2579 17981458
2. Why Most Published Research Findings Are False. JPA Ioannidis, PLoS Medicine 2005 2 124 10.1371/journal.pmed.0020124 16060722
3. Microarrays and molecular research: noise discovery. JP Ioannidis, The Lancet 2005 365 488 492 10.1016/S0140-6736(05)17866-0 (Pubitemid 40215731)
4. The control of the false discovery rate in multiple testing under dependency. Y Benjamini D Yekutieli, Journal of the Royal Statistical Society B 1995 29 1165 1188
5. A unified approach to false discovery rate estimation. K Strimmer, BMC Bioinformatics 2008 9 303 10.1186/1471-2105-9-303 18613966
6. Selection bias in gene extraction in tumour classification on basis of microarray gene expression data. C Ambroise GJ McLachlan, Proceedings of the National Academy of Science 2002 99 6562 6566 10.1073/pnas.102102699 (Pubitemid 34526172)
7. Pitfalls in the Use of DNA Microarray Data for Diagnostic and Prognostic Classification. R Simon MD Radmacher K Dobbin LM McShane, Journal of the National Cancer Institute 2003 95 14 18 12509396 (Pubitemid 36124291)
8. Predictive ability of DNA microarrays for cancer outcomes and correlates: an empirical assessment. EE Ntzani JPA Ioannidis, The Lancet 2003 362 1439 1444 10.1016/S0140-6736(03)14686-7 (Pubitemid 37352879)
9. WilcoxCV: An R package for fast variable selection in cross-validation. AL Boulesteix, Bioinformatics 2007 23 1702 1704 10.1093/bioinformatics/btm162 17495999
10. Classification based upon gene expression data: bias and precision of error rates. IA Wood PM Visscher KL Mengersen, Bioinformatics 2007 23 1363 1370 10.1093/bioinformatics/btm117 17392326
11. Regression shrinkage and selection via the LASSO. R Tibshirani, Journal of the Royal Statistical Society B 1996 58 267 288
12. Bias in error estimation when using cross-validation for model selection. S Varma R Simon, BMC Bioinformatics 2006 7 91 10.1186/1471-2105-7-91 16504092
13. Evaluating microarray-based classifiers: an overview. AL Boulesteix C Strobl T Augustin M Daumer, Cancer Informatics 2008 6 77 97 19259405
14. Critical Review of Published Microarray Studies for Cancer Outcome and Guidelines on Statistical Analysis and Reporting. A Dupuy R Simon, Journal of the National Cancer Institute 2007 99 147 157 10.1093/jnci/djk018 17227998
15. CMA (Classiciation for MicroArrays). M Slawski AL Boulesteix, Bioconductor Package version 0.8.5 2008
16. CMA - A comprehensive Bioconductor package for supervised classification with high dimensional data. M Slawski M Daumer AL Boulesteix, BMC Bioinformatics 2008 9 439 10.1186/1471-2105-9-439 18925941
17. Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. U Alon N Barkai DA Notterman K Gish S Ybarra D Mack A Levine, Proceedings of the National Academy of Sciences 1999 96 6745 6750 10.1073/pnas.96.12.6745 (Pubitemid 29274954)
18. Gene expression correlates of clinical prostate cancer behavior. D Singh PG Febbo K Ross DG Jackson J Manola C Ladd P Tamayo AA Renshaw AV D'Amico JP Richie ES Lander M Loda PW Kantoff TR Golub WR Sellers, Cancer Cell 2002 1 203 209 10.1016/S1535-6108(02)00030-2 12086878
19. Comparison of discrimination methods for the classification of tumors using gene expression data. S Dudoit J Fridlyand TP Speed, Journal of the American Statistical Association 2002 97 77 87 10.1198/016214502753479248
20. T Hastie R Tibshirani JH Friedman, The elements of statistical learning New York: Springer-Verlag 2001
21. Partial Least Squares: A versatile tool for the analysis of high-dimensional genomic data. AL Boulesteix K Strimmer, Briefings in Bioinformatics 2007 8 32 44 10.1093/bib/bbl016 16772269
22. PLS dimension reduction for classification with microarray data. AL Boulesteix, Statistical Applications in Genetics and Molecular Biology 2004 3 33 10.2202/1544-6115.1075
23. Random Forests. L Breiman, Machine Learning 2001 45 5 32 10.1023/A:1010933404324 (Pubitemid 32933532)
24. Gene selection and classification of microarray data using random forests. R Diaz-Uriarte SA de Andrés, BMC Bioinformatics 2006 7 3 10.1186/1471-2105-7-3 16398926
25. VN Vapnik, The nature of statistical learning theory New York: Springer 1995
26. Diagnosis of multiple cancer types by shrunken centroids of gene expression. R Tibshirani T Hastie B Narasimhan G Chu, Proceedings of the National Academy of Sciences 2002 99 6567 6572 10.1073/pnas.082099299 (Pubitemid 34526173)
27. Regularization and variable selection via the elastic net. H Zou T Hastie, Journal of the Royal Statistical Society B 2005 67 301 320 10.1111/j.1467-9868.2005.00503.x (Pubitemid 40465877)
28. Boosting algorithms: regularization, prediction and model fitting (with discussion). P Bühlmann T Hothorn, Statistical Science 2007 22 477 505 10.1214/07-STS242
29. Reader's reaction to 'Dimension reduction for classification with gene expression microarray data' by Dai et al (2006)'. AL Boulesteix, Statistical Applications in Genetics and Molecular Biology 2006 5 16 10.2202/1544-6115.1226
30. Linear models and empirical Bayes methods for assessing differential expression in microarray experiments. G Smyth, Statistical Applications in Genetics and Molecular Biology 2004 3 3 10.2202/1544-6115.1027
31. A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis. A Statnikov CF Aliferis I Tsamardinos D Hardin S Levy, Bioinformatics 2005 21 631 643 10.1093/ bioinformatics/bti033 15374862
32. Prediction error estimation: a comparison of resampling methods. A Molinaro R Simon RM Pfeiffer, Bioinformatics 2005 21 3301 3307 10.1093/bioinformatics/bti499 15905277
33. An extensive comparison of recent classification tools applied to microarray data. J Lee J Lee M Park S Song, Computational Statistics and Data Analysis 2005 48 869 885 10.1016/j.csda.2004.03.017
34. Reducing the probability of false positive research findings by pre-publication validation - Experience with a large multiple sclerosis database. M Daumer U Held K Ickstadt M Heinz S Schach G Ebers, BMC Medical Research Methodology 2008 8 18 10.1186/1471-2288-8-18 18402689
35. The normal fetal heart rate study: Analysis plan. M Daumer M Scholz AL Boulesteix SP von Steinburg S Schiermeier W Hatzmann KTM Schneider, Nature Precedings 2007 http://precedings.nature.com/documents/980/version/2
36. MCRestimate: Misclassification error estimation with cross-validation. M Ruschhaupt U Mansmann P Warnat W Huber A Benner, R package version 1.10.2 2007
37. Development and Validation of Therapeutically Relevant Multi-Gene Biomarker Classifiers. R Simon, Journal of the National Cancer Institute 2006 97 866 867
38. Validation and Clinical Utility of a 70-Gene Prognostic Signature for Women With Node-Negative Breast Cancer. M Buyse S Loi L van't Veer., et al. Journal of the National Cancer Institute 2006 98 1183 1192 16954471
39. Over-optimism in bioinformatics research. AL Boulesteix, Bioinformatics 2010 in press