Cross-validation and hypothesis testing in neuroimaging: An irenic comment on the exchange between Friston and Lindquist et al.

NeuroImage

Volumn 116, Issue , 2015, Pages 248-254

  Reiss, Philip T ^a,b  

^a NEW YORK UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b NATHAN S KLINE INSTITUTE FOR PSYCHIATRIC RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BOLD SIGNAL; DATA ANALYSIS; HUMAN; IMAGE ANALYSIS; IMAGING SYSTEM; MATHEMATICAL PARAMETERS; MAXIMUM LIKELIHOOD METHOD; MEASUREMENT ACCURACY; MEASUREMENT ERROR; NEUROIMAGING; NOTE; NULL HYPOTHESIS; PRIORITY JOURNAL; RADIOLOGICAL PROCEDURES; STATISTICAL MODEL; VALIDATION STUDY; METHODOLOGY; PEER REVIEW; PROCEDURES; STATISTICS;

NEUROIMAGING; PEER REVIEW, RESEARCH; RESEARCH DESIGN; STATISTICS AS TOPIC;

EID: 84941744520     PISSN: 10538119     EISSN: 10959572     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2015.04.032     Document Type: Note

References (48)

1. Abramovich F., Ritov Y. Statistical Theory: A Concise Introduction 2013, CRC Press, Boca Raton, FL.
2. Akaike H. Information theory and an extension of the maximum likelihood principle. Second International Symposium on Information Theory, Akademiai Kiado 1973, 267-281.
3. Arlot S., Celisse A. A survey of cross-validation procedures for model selection. Stat. Surv. 2010, 4:40-79.
4. Boulesteix A.L., Sauerbrei W. Added predictive value of high-throughput molecular data to clinical data and its validation. Brief. Bioinform. 2011, 12:215-229.
5. Breiman L. Statistical modeling: the two cultures (with discussion). Stat. Sci. 2001, 16:199-231.
6. Brown M.R.G., Sidhu G.S., Greiner R., Asgarian N., Bastani M., Silverstone P.H., Greenshaw A.J., Dursun S.M. ADHD-200 global competition: diagnosing ADHD using personal characteristic data can outperform resting state fMRI measurements. Front. Syst. Neurosci. 2012, 6.
7. Cohen J. Statistical Power Analysis for the Behavioral Sciences 1988, Lawrence Erlbaum Associates, Hillsdale, NJ. 2nd ed.
8. Cohen J.R., Asarnow R.F., Sabb F.W., Bilder R.M., Bookheimer S.Y., Knowlton B.J., Poldrack R.A. Decoding continuous behavioral variables from neuroimaging data. Front. Neurosci. 2011, 5.
9. Crainiceanu C.M., Ruppert D. Likelihood ratio tests in linear mixed models with one variance component. J. R. Stat. Soc. Ser. B 2004, 66:165-185.
10. Eloyan A., Muschelli J., Nebel M., Liu H., Han F., Zhao T., Barber A., Joel S., Pekar J., Mostofsky S., Caffo B. Automated diagnoses of attention deficit hyperactive disorder using magnetic resonance imaging. Front. Syst. Neurosci. 2012, 6.
11. Friedman J., Hastie T., Tibshirani R. Regularization paths for generalized linear models via coordinate descent. J. Stat. Softw. 2010, 33:1.
12. Friston K. Ten ironic rules for non-statistical reviewers. NeuroImage 2012, 61:1300-1310.
13. Friston K. Sample size and the fallacies of classical inference. NeuroImage 2013, 81:503-504.
14. Friston K.J., Penny W., Phillips C., Kiebel S., Hinton G., Ashburner J. Classical and Bayesian inference in neuroimaging: theory. NeuroImage 2002, 16:465-483.
15. Friston K.J., Ashburner J., Kiebel S.J., Nichols T., Penny W. Statistical Parametric Mapping: The Analysis of Functional Brain Images 2007, Academic Press, London.
16. Goeman J.J., Van De Geer S.A., Van Houwelingen H.C. Testing against a high dimensional alternative. J. R. Stat. Soc. Ser. B 2006, 68:477-493.
17. Goldfine A.M., Bardin J.C., Noirhomme Q., Fins J.J., Schiff N.D., Victor J.D. Reanalysis of "Bedside detection of awareness in the vegetative state: a cohort study". Lancet 2013, 381:289-291.
18. Golland P., Fischl B. Permutation tests for classification: towards statistical significance in image-based studies. Information Processing in Medical Imaging: Proceedings of the 18th International Conference 2003, 330-341. Springer, Berlin. C.J. Taylor, J.A. Noble (Eds.).
19. Golub T.R., Slonim D.K., Tamayo P., Huard C., Gaasenbeek M., Mesirov J.P., Coller H., Loh M.L., Downing J.R., Caligiuri M.A., Bloomfield C.D., Lander E.S. Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 1999, 286:531-537.
20. Hastie T.J., Tibshirani R.J., Friedman J.H. The Elements of Statistical Learning: Data Mining, Inference, and Prediction 2009, Springer, New York. 2nd ed.
21. Hoeffding W. The large-sample power of tests based on permutations of observations. Ann. Math. Stat. 1952, 23:169-192.
22. Hsing T., Attoor S., Dougherty E. Relation between permutation-test p values and classifier error estimates. Mach. Learn. 2003, 52:11-30.
23. Ingre M. Why small low-powered studies are worse than large high-powered studies and how to protect against "trivial" findings in research: comment on Friston (2012). NeuroImage 2013, 81:496-498.
24. Kass R.E., Raftery A.E. Bayes factors. J. Am. Stat. Assoc. 1995, 90:773-795.
25. Lehmann E.L., Romano J.P. Testing Statistical Hypotheses 2005, Springer, New York. 3rd ed.
26. Lindquist M.A. Functional causal mediation analysis with an application to brain connectivity. J. Am. Stat. Assoc. 2012, 107:1297-1309.
27. Lindquist M.A., Caffo B., Crainiceanu C. Ironing out the statistical wrinkles in "ten ironic rules". NeuroImage 2013, 81:499-502.
28. Lockhart R., Taylor J., Tibshirani R.J., Tibshirani R. A significance test for the lasso. Ann. Stat. 2014, 42:413-468.
29. Neyman J., Pearson E.S. On the problem of the most efficient tests of statistical hypotheses. Philos. Trans. R. Soc. Lond. Ser. A 1933, 231:289-337.
30. Nichols T.E., Holmes A.P. Nonparametric permutation tests for functional neuroimaging: a primer with examples. Hum. Brain Mapp. 2001, 15:1-25.
31. Noirhomme Q., Lesenfants D., Gomez F., Soddu A., Schrouff J., Garraux G., Luxen A., Phillips C., Laureys S. Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions. Neuroimage Clin. 2014, 4:687-694.
32. Ojala M., Garriga G. Permutation tests for studying classifier performance. J. Mach. Learn. Res. 2010, 11:1833-1863.
33. Park T., Casella G. The Bayesian Lasso. J. Am. Stat. Assoc. 2008, 103:681-686.
34. Penny W. Comparing dynamic causal models using AIC, BIC and free energy. NeuroImage 2012, 59:319-330.
35. Pepe M.S., Kerr K.F., Longton G., Wang Z. Testing for improvement in prediction model performance. Stat. Med. 2013, 32:1467-1482.
36. Phipson B., Smyth G.K. Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn. Stat. Appl. Genet. Mol. Biol. 2010, 9.
37. R Core Team R: A Language and Environment for Statistical Computing 2014, R Foundation for Statistical Computing, Vienna, Austria, (URL: http://www.R-project.org/).
38. Reiss P.T., Ogden R.T. Smoothing parameter selection for a class of semiparametric linear models. J. R. Stat. Soc. Ser. B 2009, 71:505-523.
39. Reiss P.T., Huo L., Zhao Y., Kelly C., Ogden R.T. Wavelet-domain regression and predictive inference in psychiatric neuroimaging. Ann. Appl. Stat. 2015, (in press).
40. Robinson J. The large-sample power of permutation tests for randomization models. Ann. Stat. 1973, 1:291-296.
41. Ruppert D., Wand M.P., Carroll R.J. Semiparametric Regression 2003, Cambridge University Press, New York.
42. Simon R., Radmacher M.D., Dobbin K., McShane L.M. Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification. J. Natl. Cancer Inst. 2003, 95:14-18.
43. Stone M. Cross-validatory choice and assessment of statistical predictions (with discussion). J. R. Stat. Soc. Ser. B 1974, 36:111-147.
44. Tibshirani R. Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B 1996, 58:267-288.
45. van de Wiel M.A., Berkhof J., van Wieringen W.N. Testing the prediction error difference between 2 predictors. Biostatistics 2009, 10:550-560.
46. Wilks S.S. The large-sample distribution of the likelihood ratio for testing composite hypotheses. Ann. Math. Stat. 1938, 9:60-62.
47. Winkler A.M., Ridgway G.R., Webster M.A., Smith S.M., Nichols T.E. Permutation inference for the general linear model. NeuroImage 2014, 92:381-397.
48. Wood S.N. Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models. J. R. Stat. Soc. Ser. B 2011, 73:3-36.