Towards evidence-based computational statistics: Lessons from clinical research on the role and design of real-data benchmark studies

BMC Medical Research Methodology

Volumn 17, Issue 1, 2017, Pages

  Boulesteix, Anne Laure ^a   Wilson, Rory ^a   Hapfelmeier, Alexander ^b  

^a UNIVERSITY OF MUNICH   (Germany)

^b TECHNICAL UNIVERSITY OF MUNICH   (Germany)

Author keywords

Clinical trial; Comparison study; Good practice; Method evaluation

Indexed keywords

ANIMAL; BENCHMARKING; CLINICAL TRIAL (TOPIC); EVIDENCE BASED MEDICINE; HUMAN; INFORMATION PROCESSING; MEDICAL RESEARCH; STATISTICS;

ANIMALS; BENCHMARKING; BIOMEDICAL RESEARCH; CLINICAL TRIALS AS TOPIC; DATASETS AS TOPIC; EVIDENCE-BASED MEDICINE; HUMANS; STATISTICS AS TOPIC;

EID: 85029093683     PISSN: None     EISSN: 14712288     Source Type: Journal    
DOI: 10.1186/s12874-017-0417-2     Document Type: Article

References (35)

1. Greenhalgh T, Howick J, Maskrey N. Evidence based medicine: a movement in crisis?Br Med J. 2014; 348:3725. doi: 10.1136/bmj.g3725.
2. Sauerbrei W, Abrahamowicz M, Altman DG, Le Cessie S, Carpenter J. Strengthening analytical thinking for observational studies: the STRATOS initiative. Stat Med. 2014; 33(30):5413-432. doi: 10.1002/sim.6265.
3. Lichman M. UCI Machine Learning Repository. 2013. http://archive.ics.uci.edu/ml. Accessed 6 Sept 2017.
4. Vanschoren J, van Rijn JN, Bischl B, Torgo L. OpenML: Networked science in machine learning. SIGKDD Explor. 2013; 15(2):49-60. doi: 10.1145/2641190.2641198.
5. In: Guyon I, Cawley G, Dror G, (eds).Hands-On Pattern Recognition: Challenges in Machine Learning, Volume 1: MICROTOME PUB, Brookline, Massachusetts; 2011. https://www.amazon.com/Hands-Pattern-Recognition-Challenges-Learning/dp/0971977712.
6. Boulesteix AL, Hornung R, Sauerbrei W. On fishing for significance and statistician's degree of freedom in the era of big molecular data In: Wernecke J, Pietsch W, Otte M, editors. Berechenbarkeit der Welt? Philosophie und Wissenschaft Im Zeitalter Von Big Data. Springer VS: 2017. p. 155-170.
7. Silberzahn R, Uhlmann EL. Crowdsourced research: Many hands make tight work. Nature. 2015; 526(7572):189-91. doi: 10.1038/526189a.
8. Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005; 2(8):124. doi: 10.1371/journal.pmed.0020124.
9. Boulesteix AL, Hable R, Lauer S, Eugster MJA. A statistical framework for hypothesis testing in real data comparison studies. Am Stat. 2015; 69(3):201-12. doi: 10.1080/00031305.2015.1005128.
10. Boulesteix AL, Strobl C, Augustin T, Daumer M. Evaluating microarray-based classifiers: an overview. Cancer Informat. 2008; 6:77-97.
11. Dudoit S, Fridlyand J, Speed TP. Comparison of discrimination methods for the classification of tumors using gene expression data. J Am Stat Assoc. 2002; 97(457):77-87.
12. Romualdi C, Campanaro S, Campagna D, Celegato B, Cannata N, Toppo S, Valle G, Lanfranchi G. Pattern recognition in gene expression profiling using dna array: a comparative study of different statistical methods applied to cancer classification. Hum Mol Genet. 2003; 12:823-36.
13. Li T, Zhang C, Ogihara M. A comparative study of feature selection and multiclass classification methods for tissue classification based on gene expression. Bioinformatics. 2004; 20:2429-37.
14. Statnikov A, Aliferis CF, Tsamardinos I, Hardin D, Levy S. A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis. Bioinformatics. 2005; 21(5):631-43.
15. Lee JW, Lee JB, Park M, Song SH. An extensive comparison of recent classification tools applied to microarray data. Comput Stat Data Anal. 2005; 48(4):869-85.
16. Huang X, Pan W, Grindle S, Han X, Chen Y, Park S, Miller L, Hall J. A comparative study of discriminating human heart failure etiology using gene expression profiles. BMC Bioinforma. 2005; 6:205.
17. Statnikov A, Wang L, Aliferis C. A comprehensive comparison of random forests and support vector machines for microarray-based cancer classification. BMC Bioinforma. 2008; 9:319.
18. de Souza BF, de Carvalho A, Soares C. A comprehensive comparison of ml algorithms for gene expression data classification. In: The 2010 International Joint Conference on Neural Networks (IJCNN). IEEE: 2010. p. 1-8. http://ieeexplore.ieee.org/document/5596651/?reload=true.
19. Boulesteix AL, Lauer S, Eugster MJA. A plea for neutral comparison studies in computational sciences. PLoS ONE. 2013; 8:61562. doi: 10.1371/journal.pone.0061562.
20. Hornung R, Boulesteix AL, Causeur D. Combining location-and-scale batch effect adjustment with data cleaning by latent factor adjustment. BMC Bioinforma. 2016; 17:27. doi: 10.1186/s12859-015-0870-z.
21. Fernández-Delgado M, Cernadas E, Barro S, Amorim D. Do we need hundreds of classifiers to solve real world classification problems. J Mach Learn Res. 2014; 15(1):3133-81.
22. Yousefi MR, Hua J, Sima C, Dougherty ER. Reporting bias when using real data sets to analyze classification performance. Bioinformatics. 2010; 26(1):68-76. doi: 10.1093/bioinformatics/btp605.
23. Rennie D. CONSORT revised - improving the reporting of randomized trials. JAMA. 2001; 285(15):2006-7. doi: 10.1001/jama.285.15.2006.
24. Boulesteix AL, Stierle V, Hapfelmeier A. Publication bias in methodological computational research. Cancer Informat. 2015; 14(Suppl 5):11-19. doi: 10.4137/CIN.S30747.
25. De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJP, et al. Clinical trial registration: a statement from the international committee of medical journal editors. N Engl J Med. 2004; 351(12):1250-51.
26. Chambers CD. Registered reports: A new publishing initiative at Cortex [editorial]. Cortex. 2013; 49(3):609-10.
27. Jelizarow M, Guillemot V, Tenenhaus A, Strimmer K, Boulesteix AL. Over-optimism in bioinformatics: an illustration. Bioinformatics. 2010; 26(16):1990-8. doi: 10.1093/bioinformatics/btq323.
28. Binder H, Sauerbrei W, Royston P. Comparison between splines and fractional polynomials for multivariable model building with continuous covariates: a simulation study with continuous response. Stat Med. 2013; 32(13):2262-77.
29. Bischl B, Schiffner J, Weihs C. Benchmarking local classification methods. Comput Stat. 2013; 28(6):2599-619. doi: 10.1007/s00180-013-0420-y.
30. Duin RP. A note on comparing classifiers. Pattern Recogn Lett. 1996; 17(5):529-36.
31. Canadian Task Force on the Periodic Health Examination. The periodic health examination. Can Med Assoc J. 1979; 121(9):1193-54.
32. Boulesteix AL. On representative and illustrative comparisons with real data in bioinformatics: response to the letter to the editor by smith et al. Bioinformatics. 2013; 29(20):2664-6.
33. Jamain A, Hand DJ. Mining supervised classification performance studies: A meta-analytic investigation. J Classif. 2008; 25(1):87-112. doi: 10.1007/s00357-008-9003-y.
34. Sargent DJ. Comparison of artificial neural networks with other statistical approaches: results from medical data sets. Cancer. 2001; 91:1636-42.
35. Couronné R, Probst P, Boulesteix AL. Random forest versus logistic regression: a large-scale benchmark experiment. 2017. Technical Report 205, Department of Statistics, LMU Munich.