Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions

NeuroImage: Clinical

Volumn 4, Issue , 2014, Pages 687-694

  Noirhomme, Quentin ^a,b   Lesenfants, Damien ^a,b   Gomez, Francisco ^c   Soddu, Andrea ^d   Schrouff, Jessica ^a,e   Garraux, Gaëtan ^a   Luxen, André ^a   Phillips, Christophe ^a   Laureys, Steven ^a,b  

^a UNIVERSITY OF LIÈGE   (Belgium)

^b UNIVERSITY OF LIÈGE   (Belgium)

^c UNIVERSIDAD CENTRAL   (Colombia)

^d UNIVERSITY OF WESTERN ONTARIO   (Canada)

^e STANFORD UNIVERSITY   (United States)

Author keywords

binomial; classification; cross validation; permutation test

Indexed keywords

ADULT; AGED; ARTICLE; BINOMIAL DISTRIBUTION; BOLD SIGNAL; BRAIN COMPUTER INTERFACE; CLINICAL ARTICLE; CONSCIOUSNESS; CONTROLLED STUDY; DIAGNOSTIC ACCURACY; DIAGNOSTIC TEST ACCURACY STUDY; DISEASE CLASSIFICATION; ELECTROENCEPHALOGRAM; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; GAIT; HUMAN; IDIOPATHIC DISEASE; MALE; MINIMALLY CONSCIOUS STATE; PARKINSON DISEASE; PRIORITY JOURNAL; SUPPORT VECTOR MACHINE; CONSCIOUSNESS DISORDER; CROSS VALIDATION; FUNCTIONAL NEUROIMAGING; VALIDATION STUDY; BRAIN; BRAIN INJURIES; COMPUTER SIMULATION; MIDDLE AGED; PATHOLOGY; STATISTICAL BIAS; STATISTICAL MODEL; YOUNG ADULT;

ADULT; AGED; BIAS (EPIDEMIOLOGY); BRAIN; BRAIN INJURIES; BRAIN-COMPUTER INTERFACES; COMPUTER SIMULATION; HUMANS; MIDDLE AGED; MODELS, STATISTICAL; YOUNG ADULT;

EID: 84899428996     PISSN: None     EISSN: 22131582     Source Type: Journal    
DOI: 10.1016/j.nicl.2014.04.004     Document Type: Article

References (48)

1. Benjamini Y., Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing Journal of the Royal Statistical Society B 57 1995 289 300
2. Berrar D., Bradbury I. Avoiding model selection bias in small-sample genomic datasets Bioinformatics (Oxford, England) 22 10 2006 1245 1250 10.1093/bioinformatics/btl066 16500931
3. Billinger M., Daly I. Is it significant? B.Z. Allison S. Dunne R. Leeb R. Millan Jdel A. Nijholt Guidelines for Reporting BCI Performance. Towards Practical Brain-Computer Interfaces 2013 Springer Berlin, Heidelberg 333 354
4. Burges C. A tutorial on support vector machines for pattern recognition Data Mining and Knowledge Discovery 2 1998 121 167 10.1023/A:1009715923555
5. Chang C.-C., Lin C.-J. LIBSVM: a library for support vector machines ACM Transactions on Intelligent Systems and Technology 2 3 2011 27:21 27:27
6. Cremers J., D'Ostilio K. Brain activation pattern related to gait disturbances in Parkinson's disease Movement Disorders: Official Journal of the Movement Disorder Society 27 12 2012 1498 1505 10.1002/mds.25139 23008169
7. Cruse D., Chennu S. Bedside detection of awareness in the vegetative state: a cohort study Lancet 378 9809 2011 2088 2094 10.1016/S0140-6736(11) 61224-5 22078855
8. Donchin E., Spencer K.M. The mental prosthesis: assessing the speed of a P300-based brain-computer interface IEEE Transactions on Rehabilitation Engineering: A Publication of the IEEE Engineering in Medicine and Biology Society 8 2 2000 174 179 10.1109/86.847808 10896179
9. Efron B., Tibshirani R. Improvements on cross-validation: the.632+ bootstrap method Journal of the American Statistical Association 92 438 1997 548 560 10.1080/01621459.1997.1047400710.2307/2965703
10. Etzel J.A., Gazzola V. An introduction to anatomical ROI-based fMRI classification analysis Brain Research 1282 2009 114 125 10.1016/j.brainres. 2009.05.090 19505449
11. Farwell L.A., Donchin E. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials Electroencephalography and Clinical Neurophysiology 70 6 1988 510 523 10.1016/0013-4694(88)90149-6 2461285
12. Focke N.K., Helms G. Individual voxel-based subtype prediction can differentiate progressive supranuclear palsy from idiopathic Parkinson syndrome and healthy controls Human Brain Mapping 32 11 2011 1905 1915 10.1002/hbm.21161 21246668
13. Furdea A., Halder S. An auditory oddball (P300) spelling system for brain-computer interfaces Psychophysiology 46 3 2009 617 625 10.1111/j.1469-8986.2008.00783.x 19170946
14. Galanaud D., Perlbarg V. Assessment of white matter injury and outcome in severe brain trauma: a prospective multicenter cohort Anesthesiology 117 6 2012 1300 1310 10.1097/ALN.0b013e3182755558 23135261
15. Garraux G., Phillips C. Multiclass classification of FDG PET scans for the distinction between Parkinson's disease and atypical parkinsonian syndromes NeuroImage: Clinical 2 2013 883 893 10.1016/j.nicl.2013.06.004 24179839
16. Goldfine A.M., Bardin J.C. Reanalysis of "bedside detection of awareness in the vegetative state: a cohort study" Lancet 381 9863 2013 289 291 10.1016/S0140-6736(13)60125-7 23351802
17. Golub T.R., Slonim D.K. Molecular classification of cancer: class discovery and class prediction by gene expression monitoring Science (New York, N.Y.) 286 5439 1999 531 537 10.1126/science.286.5439.531 10521349
18. Good P. Permutation, Parametric and Bootstrap Tests of Hypotheses 2005 Springer United States of America
19. Howell D.C. Statistical Methods for Psychology 2012 Wadsworth
20. Hughes A.J., Daniel S.E. Accuracy of clinical diagnosis of idiopathic Parkinson's disease: a clinico-pathological study of 100 cases Journal of Neurology, Neurosurgery, and Psychiatry 55 3 1992 181 184 10.1136/jnnp.55.3.181 1564476
21. Kohavi R. A study of cross-validation and bootstrap for accuracy estimation and model selection International Joint Conference on Artificial Intelligence 1995
22. Krusienski D.J., Sellers E.W. A comparison of classification techniques for the P300 Speller Journal of Neural Engineering 3 4 2006 299 305 10.1088/1741-2560/3/4/007 17124334
23. Kubler A., Birbaumer N. Brain-computer interfaces and communication in paralysis: extinction of goal directed thinking in completely paralysed patients? Clinical Neurophysiology: Official Journal of the International Federation of Clinical Neurophysiology 119 11 2008 2658 2666 10.1016/j.clinph.2008.06.019 18824406
24. Laureys S., Schiff N.D. Coma and consciousness: paradigms (re)framed by neuroimaging Neuroimage 61 2 2012 478 491 10.1016/j.neuroimage.2011.12.041 22227888
25. Lemm S., Blankertz B. Introduction to machine learning for brain imaging Neuroimage 56 2 2011 387 399 10.1016/j.neuroimage.2010.11.004 21172442
26. Lule D., Noirhomme Q. Probing command following in patients with disorders of consciousness using a brain-computer interface Clinical Neurophysiology: Official Journal of the International Federation of Clinical Neurophysiology 124 1 2013 101 106 10.1016/j.clinph.2012.04.030 22920562
27. Luyt C.E., Galanaud D. Diffusion tensor imaging to predict long-term outcome after cardiac arrest: a bicentric pilot study Anesthesiology 117 6 2012 1311 1321 10.1097/ALN.0b013e318275148c 23135257
28. Maillet A., Pollak P. Imaging gait disorders in parkinsonism: a review Journal of Neurology, Neurosurgery, and Psychiatry 83 10 2012 986 993 10.1136/jnnp-2012-302461 22773859
29. Maris E., Oostenveld R. Nonparametric statistical testing of EEG- and MEG-data Journal of Neuroscience Methods 164 1 2007 177 190 10.1016/j.jneumeth. 2007.03.024 17517438
30. Martin J.K., Hirschberg D.S. Small Sample Statistics for Classification Error Rates II: Confidence Intervals and Significance Tests 1996 Department of Information and Computer Science, University of California, Irvine CA.
31. Mukherjee S., Golland P. Permutation Tests for Classification 2003 Massachusetts Institute of Technology Cambridge, MA 22
32. Müller-Putz G.R., Scherer R. Better than random? A closer look on BCI results International. Journal of Bioelectromagnetism. 10 1 2008 52 55
33. Nichols T.E., Holmes A.P. Nonparametric permutation tests for functional neuroimaging: a primer with examples Human Brain Mapping 15 1 2002 1 25 10.1002/hbm.1058 11747097
34. Ojala M., Garriga G.C. Permutation tests for studying classifier performance Journal of Machine Learning Research 11 2010 1833 1863
35. Orru G., Pettersson-Yeo W. Using support vector machine to identify imaging biomarkers of neurological and psychiatric disease: a critical review Neuroscience and Biobehavioral Reviews 36 4 2012 1140 1152 10.1016/j.neubiorev. 2012.01.004 22305994
36. Pereira F., Botvinick M. Information mapping with pattern classifiers: a comparative study Neuroimage 56 2 2011 476 496 10.1016/j.neuroimage.2010.05.026 20488249
37. Pereira F., Detre G. Generating text from functional brain images Frontiers in Human Neuroscience 5 2011 72 21927602
38. Pereira F., Mitchell T. Machine learning classifiers and fMRI: a tutorial overview NeuroImage 45 1 Suppl. 2009 S199 S209 10.1016/j.neuroimage.2008.11.007 19070668
39. Phillips C.L., Bruno M.A. "Relevance vector machine" consciousness classifier applied to cerebral metabolism of vegetative and locked-in patients Neuroimage 56 2 2011 797 808 10.1016/j.neuroimage.2010.05.083 20570741
40. Picard R.R., Cook R.D. Cross-validation of regression models Journal of the American Statistical Association 79 1984 575 583 10.1080/01621459.1984. 10478083
41. Schalk G., McFarland D.J. BCI2000: A general-purpose brain-computer interface (BCI) system IEEE Transactions on Biomedical Engineering 51 6 2004 1034 1043 10.1109/TBME.2004.827072 15188875
42. Schrouff J., Cremers J. Discriminant BOLD activation patterns during mental imagery in Parkinson's disease Machine Learning and Interpretation in NeuroImaging workshop at NIPS 2012 NIPS South Lake Tahoe, United States of America 8
43. Schrouff J., Cremers J. Localizing and comparing weight maps generated from linear kernel machine learning models 3rd Workshop on Pattern Recognition in NeuroImaging (PRNI 2013) 2013 IEEE Computer Society Conference Publishing Services Philadelphia, USA 4
44. Schrouff J., Rosa M.J. PRoNTo: pattern recognition for neuroimaging toolbox Neuroinformatics 11 3 2013 319 337 10.1007/s12021-013-9178-1 23417655
45. Sellers E.W., Donchin E. A P300-based brain-computer interface: initial tests by ALS patients Clinical Neurophysiology: Official Journal of the International Federation of Clinical Neurophysiology 117 3 2006 538 548 10.1016/j.clinph.2005.06.027 16461003
46. Sharbrough F., Chatrian G.-E. American Electroencephalographic Society Guidelines for Standard Electrode Position Nomenclature Journal of Clinical Neurophysiology: Official Publication of the American Electroencephalographic Society 8 1991 200 202 10.1097/00004691-199104000-00007 2050819
47. Simon R., Radmacher M.D. Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification Journal of the National Cancer Institute 95 1 2003 14 18 10.1093/jnci/95.1.14 12509396
48. Cruse D., Gantner I., Soddu A., Owen A.M. Lies, damned lies, and diagnoses: Estimating the clinical utility of assessments of covert awareness in the Vegetative State. Brain Inj: in press.