Seven pervasive statistical flaws in cognitive training interventions

Frontiers in Human Neuroscience

Volumn 10, Issue APR2016, 2016, Pages 1-17

  Moreau, David ^a   Kirk, Ian J ^a   Waldie, Karen E ^a  

^a UNIVERSITY OF AUCKLAND   (New Zealand)

Author keywords

Brain enhancement; Data analysis; Evidence based interventions; Experimental design; Intelligence; Methods; Statistics; Working memory training

Indexed keywords

ARTICLE; COGNITION; EVIDENCE BASED MEDICINE; HUMAN; MONTE CARLO METHOD; NEUROSCIENCE; PSYCHOLOGY; PUBLISHING; SAMPLING ERROR; SCORING SYSTEM; TASK PERFORMANCE; WORKING MEMORY;

EID: 84964746455     PISSN: None     EISSN: 16625161     Source Type: Journal    
DOI: 10.3389/fnhum.2016.00153     Document Type: Article

References (116)

1. Addelman, S. (1969). The generalized randomized block design. Am. Stat. 23, 35-36. doi: 10.2307/2681737
2. Allison, D. B., Gorman, B. S., and Primavera, L. H. (1993). Some of the most common questions asked of statistical consultants: our favorite responses and recommended readings. Genet. Soc. Gen. Psychol. Monogr. 119, 155-185.
3. Andrews, M., and Baguley, T. (2013). Prior approval: the growth of Bayesian methods in psychology. Br. J. Math. Stat. Psychol. 66, 1-7. doi: 10.1111/bmsp.12004
4. Anguera, J. A., and Gazzaley, A. (2015). Video games, cognitive exercises and the enhancement of cognitive abilities. Curr. Opin. Behav. Sci. 4, 160-165. doi: 10.1016/j.cobeha.2015.06.002
5. Aoyama, H. (1962). Stratified random sampling with optimum allocation for multivariate population. Ann. Inst. Stat. Math. 14, 251-258. doi: 10.1007/bf02868647
6. Au, J., Sheehan, E., Tsai, N., Duncan, G. J., Buschkuehl, M., and Jaeggi, S. M. (2015). Improving fluid intelligence with training on working memory: a metaanalysis. Psychon. Bull. Rev. 22, 366-377. doi: 10.3758/s13423-014-0699-x
7. Auguie, B. (2012). GridExtra: Functions in Grid Graphics. (R package version 0.9.1).
8. Bakan, D. (1953). Learning and the principle of inverse probability. Psychol. Rev. 60, 360-370. doi: 10.1037/h0055248
9. Bakan, D. (1966). The test of significance in psychological research. Psychol. Bull. 66, 423-437. doi: 10.1037/h0020412
10. Baker, M. (2015). First results from psychology’s largest reproducibility test. Nature doi: 10.1038/nature.2015.17433 [Epub ahead of print].
11. Bavelier, D., Green, C. S., Pouget, A., and Schrater, P. (2012). Brain plasticity through the life span: learning to learn and action video games. Annu. Rev. Neurosci. 35, 391-416. doi: 10.1146/annurev-neuro-060909-152832
12. Bishop, D. V. M., and Thompson, P. A. (2016). Problems in using p-curve analysis and text-mining to detect rate of p-hacking and evidential value. PeerJ 4: e1715. doi: 10.7717/peerj.1715
13. Boot, W. R., Simons, D. J., Stothart, C., and Stutts, C. (2013). The pervasive problem with placebos in psychology: why active control groups are not sufficient to rule out placebo effects. Perspect. Psychol. Sci. 8, 445-454. doi: 10.1177/1745691613491271
14. Bossaer, J. B., Gray, J. A., Miller, S. E., Enck, G., Gaddipati, V. C., and Enck, R. E. (2013). The use and misuse of prescription stimulants as '‘cognitive enhancers’' by students at one academic health sciences center. Acad. Med. 88, 967-971. doi: 10.1097/ACM.0b013e318294fc7b
15. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365-376. doi: 10. 1038/nrn3475
16. Campbell, D. T., and Stanley, J. (1966). Experimental and Quasiexperimental Designs for Research. Chicago: Rand McNally.
17. Carmines, E. G., and Zeller, R. A. (1979). Reliability and Validity Assessment. Thousand Oaks, CA: SAGE Publications, Inc.
18. Champely, S. (2015). Pwr: Basic Functions for Power Analysis. (R package version 1.1-2).
19. Chein, J. M., and Morrison, A. B. (2010). Expanding the mind’s workspace: training and transfer effects with a complex working memory span task. Psychon. Bull. Rev. 17, 193-199. doi: 10.3758/PBR.17.2.193
20. Cohen, J. (1983). The cost of dichotomization. Appl. Psychol. Meas. 7, 249-253. doi: 10.1177/014662168300700301
21. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. 2nd Edn. Hillsdale, NJ: Lawrence Earlbaum Associates.
22. Cohen, J. (1992a). A power primer. Psychol. Bull. 112, 155-159. doi: 10.1037/0033-2909.112.1.155
23. Cohen, J. (1992b). Statistical power analysis. Curr. Dir. Psychol. Sci. 1, 98-101. doi: 10.1111/1467-8721.ep10768783
24. Cumming, G. (2014). The new statistics: why and how. Psychol. Sci. 25, 7-29. doi: 10.1177/0956797613504966
25. Dean, C. B., and Nielsen, J. D. (2007). Generalized linear mixed models: a review and some extensions. Lifetime Data Anal. 13, 497-512. doi: 10.1007/s10985-007-9065-x
26. Dickersin, K. (1990). The existence of publication bias and risk factors for its occurrence. JAMA 263, 1385-1389. doi: 10.1001/jama.263.10.1385
27. Earp, B. D., Sandberg, A., Kahane, G., and Savulescu, J. (2014). When is diminishment a form of enhancement? Rethinking the enhancement debate in biomedical ethics. Front. Syst. Neurosci. 8: 12. doi: 10.3389/fnsys.2014.00012
28. Edwards, W., Lindman, H., and Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychol. Rev. 70, 193-242. doi: 10.1037/h0044139
29. Ellis, H. C. (1965). The Transfer of Learning. New York, NY: MacMillian.
30. Ericson, W. A. (2012). Optimum stratified sampling using prior information. J. Am. Stat. Assoc. 60, 311-750. doi: 10.2307/2283243
31. Ericsson, K. A., Krampe, R. T., and Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychol. Rev. 100, 363-406. doi: 10.1037/0033-295x.100.3.363
32. Fiedler, K. (2011). Voodoo correlations are everywhere-not only in neuroscience. Perspect. Psychol. Sci. 6, 163-171. doi: 10.1177/1745691611400237
33. Forman, S. D., Cohen, J. D., Fitzgerald, M., Eddy, W. F., Mintun, M. A., and Noll, D. C. (1995). Improved assessment of significant activation in functional magnetic resonance imaging (fMRI): use of a cluster-size threshold. Magn. Reson. Med. 33, 636-647. doi: 10.1002/mrm.1910330508
34. Francis, G. (2012). Too good to be true: publication bias in two prominent studies from experimental psychology. Psychon. Bull. Rev. 19, 151-156. doi: 10.3758/s13423-012-0227-9
35. Galton, F. (1886). Regression towards mediocrity in hereditary stature. J. Anthropol. Inst. Great Br. Irel. 15, 246-263. doi: 10.2307/2841583
36. Gelman, A., Hill, J., and Yajima, M. (2012). Why we (usually) don’t have to worry about multiple comparisons. J. Res. Educ. Eff. 5, 189-211.
37. Ghisletta, P., and McArdle, J. J. (2012). Teacher’s corner: latent curve models and latent change score models estimated in R. Struct. Equ. Modeling 19, 651-682. doi: 10.1080/10705511.2012.713275
38. Gilbert, D. T., King, G., Pettigrew, S., and Wilson, T. D. (2016). Comment on '‘Estimating the reproducibility of psychological science’'. Science 351: 1037. doi: 10.1126/science.aad7243
39. Goldacre, B. (2015). How to get all trials reported: audit, better data and individual accountability. PLoS Med. 12: e1001821. doi: 10.1371/journal.pmed.1001821
40. Graham, L., Bellert, A., Thomas, J., and Pegg, J. (2007). QuickSmart: a basic academic skills intervention for middle school students with learning difficulties. J. Learn. Disabil. 40, 410-419. doi: 10.1177/00222194070400050401
41. Green, C. S., Strobach, T., and Schubert, T. (2014). On methodological standards in training and transfer experiments. Psychol. Res. 78, 756-772. doi: 10.1007/s00426-013-0535-3
42. Hayes, T. R., Petrov, A. A., and Sederberg, P. B. (2015). Do we really become smarter when our fluid-intelligence test scores improve? Intelligence 48, 1-14. doi: 10.1016/j.intell.2014.10.005
43. Helland, T., Tjus, T., Hovden, M., Ofte, S., and Heimann, M. (2011). Effects of bottom-up and top-down intervention principles in emergent literacy in children at risk of developmental dyslexia: a longitudinal study. J. Learn. Disabil. 44, 105-122. doi: 10.1177/0022219410391188
44. Herzog, S., and Ostwald, D. (2013). Experimental biology: sometimes Bayesian statistics are better. Nature 494: 35. doi: 10.1038/494035b
45. Hurley, D. (2014). Smarter: The New Science of Building Brain Power. New York, NY: Penguin Group (USA) LLC.
46. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Med. 2: e124. doi: 10.1371/journal.pmed.0020124
47. Jaeggi, S. M., Buschkuehl, M., Jonides, J., and Shah, P. (2011). Short-and long-term benefits of cognitive training. Proc. Natl. Acad. Sci. U S A 108, 10081-10086. doi: 10.1073/pnas.1103228108
48. Jeffreys, H. (1961). The Theory of Probability. 3rd Edn. Oxford, NY: Oxford University Press.
49. Karbach, J., and Verhaeghen, P. (2014). Making working memory work: a metaanalysis of executive-control and working memory training in older adults. Psychol. Sci. 25, 2027-2037. doi: 10.1177/0956797614548725
50. Kelley, K., and Lai, K. (2012). MBESS: MBESS. (R package version 3.3.3).
51. Kimmelman, J., Mogil, J. S., and Dirnagl, U. (2014). Distinguishing between exploratory and confirmatory preclinical research will improve translation. PLoS Biol. 12: e1001863. doi: 10.1371/journal.pbio.1001863
52. Kirk, R. E. (1996). Practical significance: a concept whose time has come. Educ. Psychol. Meas. 56, 746-759. doi: 10.1177/0013164496056005002
53. Krzywinski, M., and Altman, N. (2013). Points of significance: power and sample size. Nat. Methods 10, 1139-1140. doi: 10.1038/nmeth.2738
54. Kundu, B., Sutterer, D. W., Emrich, S. M., and Postle, B. R. (2013). Strengthened effective connectivity underlies transfer of working memory training to tests of short-term memory and attention. J. Neurosci. 33, 8705-8715. doi: 10.1523/JNEUROSCI.2231-13.2013
55. Laird, N. M., and Ware, J. H. (1982). Random-effects models for longitudinal data. Biometrics 38, 963-974. doi: 10.2307/2529876
56. Lampit, A., Hallock, H., and Valenzuela, M. (2014). Computerized cognitive training in cognitively healthy older adults: a systematic review and metaanalysis of effect modifiers. PLoS Med. 11: e1001756. doi: 10.1371/journal.pmed.1001756
57. Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., and Olkin, I. (2006). The case of the misleading funnel plot. BMJ 333, 597-600. doi: 10.1136/bmj.333.7568.597
58. Leek, J. T., and Peng, R. D. (2015). Statistics: P values are just the tip of the iceberg. Nature 520: 612. doi: 10.1038/520612a
59. Loosli, S. V., Buschkuehl, M., Perrig, W. J., and Jaeggi, S. M. (2012). Working memory training improves reading processes in typically developing children. Child Neuropsychol. 18, 62-78. doi: 10.1080/09297049.2011.575772
60. Lövdén, M., Ghisletta, P., and Lindenberger, U. (2005). Social participation attenuates decline in perceptual speed in old and very old age. Psychol. Aging 20, 423-434. doi: 10.1037/0882-7974.20.3.423
61. MacCallum, R. C., Zhang, S., Preacher, K. J., and Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychol. Methods 7, 19-40. doi: 10.1037/1082-989x.7.1.19
62. McArdle, J. J. (2009). Latent variable modeling of differences and changes with longitudinal data. Annu. Rev. Psychol. 60, 577-605. doi: 10.1146/annurev.psych.60.110707.163612
63. McArdle, J. J., and Prindle, J. J. (2008). A latent change score analysis of a randomized clinical trial in reasoning training. Psychol. Aging 23, 702-719. doi: 10.1037/a0014349
64. Melby-Lervåg, M., and Hulme, C. (2013). Is working memory training effective? A meta-analytic review. Dev. Psychol. 49, 270-291. doi: 10.1037/a0028228
65. Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., et al. (1998). Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 352, 609-613. doi: 10.1016/s0140-6736(98)01085-x
66. Moreau, D. (2014a). Can brain training boost cognition? Nature 515: 492. doi: 10.1038/515492c
67. Moreau, D. (2014b). Making sense of discrepancies in working memory training experiments: a Monte Carlo simulation. Front. Syst. Neurosci. 8: 161. doi: 10.3389/fnsys.2014.00161
68. Moreau, D., and Conway, A. R. A. (2014). The case for an ecological approach to cognitive training. Trends Cogn. Sci. 18, 334-336. doi: 10.1016/j.tics.2014.03.009
69. Moreau, D., and Waldie, K. E. (2016). Developmental learning disorders: from generic interventions to individualized remediation. Front. Psychol. 6: 2053. doi: 10.3389/fpsyg.2015.02053
70. Morey, R. D., and Rouder, J. N. (2011). Bayes factor approaches for testing interval null hypotheses. Psychol. Methods 16, 406-419. doi: 10.1037/a0024377
71. Nesselroade, J. R., Stigler, S. M., and Baltes, P. B. (1980). Regression toward the mean and the study of change. Psychol. Bull. 88, 622-637. doi: 10.1037/0033-2909.88.3.622
72. Neyman, J., and Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 231, 289-337. doi: 10.1098/rsta.1933.0009
73. Noack, H., Lövdén, M., and Schmiedek, F. (2014). On the validity and generality of transfer effects in cognitive training research. Psychol. Res. 78, 773-789. doi: 10.1007/s00426-014-0564-6
74. Novick, J. M., Hussey, E., Teubner-Rhodes, S., Harbison, J. I., and Bunting, M. F. (2014). Clearing the garden-path: improving sentence processing through cognitive control training. Lang. Cogn. Neurosci. 29, 186-217. doi: 10.1080/01690965.2012.758297
75. Nuzzo, R. (2014). Scientific method: statistical errors. Nature 506, 150-152. doi: 10.1038/506150a
76. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science 349: aac4716. doi: 10.1126/science.aac4716
77. Pashler, H., and Wagenmakers, E.-J. (2012). Editors’ introduction to the special section on replicability in psychological science: a crisis of confidence? Perspect. Psychol. Sci. 7, 528-530. doi: 10.1177/1745691612465253
78. Porsdam Mann, S., and Sahakian, B. J. (2015). The increasing lifestyle use of modafinil by healthy people: safety and ethical issues. Curr. Opin. Behav. Sci. 4, 136-141. doi: 10.1016/j.cobeha.2015.05.004
79. R Core Team. (2014). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing.
80. Redick, T. S., Shipstead, Z., Harrison, T. L., Hicks, K. L., Fried, D. E., Hambrick, D. Z., et al. (2013). No evidence of intelligence improvement after working memory training: a randomized, placebo-controlled study. J. Exp. Psychol. Gen. 142, 359-379. doi: 10.1037/a0029082
81. Revelle, W. (2015). Psych: Procedures for Personality and Psychological Research. Evanston, Illinois: Northwestern University.
82. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., and Iverson, G. (2009). Bayesian tests for accepting and rejecting the null hypothesis. Psychon. Bull. Rev. 16, 225-237. doi: 10.3758/pbr.16.2.225
83. Rovine, M. J., and Molenaar, P. C. (2000). A structural modeling approach to a multilevel random coefficients model. Multivariate Behav. Res. 35, 51-88. doi: 10.1207/S15327906MBR3501_3
84. Rubinstein, R. Y., and Kroese, D. P. (2011). Simulation and the Monte Carlo Method. (Vol. 80). New York, NY: John Wiley & Sons.
85. Rudebeck, S. R., Bor, D., Ormond, A., O’Reilly, J. X., and Lee, A. C. H. (2012). A potential spatial working memory training task to improve both episodic memory and fluid intelligence. PLoS One 7: e50431. doi: 10.1371/journal.pone.0050431
86. Savage, L. J. (1954). The Foundations of Statistics (Dover Edit.). New York, NY: Wiley.
87. Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis and cumulative knowledge in psychology. Am. Psychol. 47, 1173-1181. doi: 10.1037/0003-066x.47.10.1173
88. Schubert, T., and Strobach, T. (2012). Video game experience and optimized executive control skills-On false positives and false negatives: reply to Boot and Simons (2012). Acta Psychol. 141, 278-280. doi: 10.1016/j.actpsy.2012.06.010
89. Schweizer, S., Grahn, J., Hampshire, A., Mobbs, D., and Dalgleish, T. (2013). Training the emotional brain: improving affective control through emotional working memory training. J. Neurosci. 33, 5301-5311. doi: 10.1523/JNEUROSCI.2593-12.2013
90. Shipstead, Z., Redick, T. S., and Engle, R. W. (2012). Is working memory training effective? Psychol. Bull. 138, 628-654. doi: 10.1037/a0027473
91. Simmons, J. P., Nelson, L. D., and Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359-1366. doi: 10.1177/0956797611417632
92. Simons, D. J. (2014). The value of direct replication. Perspect. Psychol. Sci. 9, 76-80. doi: 10.1177/1745691613514755
93. Simonsohn, U. (2015). Small telescopes: detectability and the evaluation of replication results. Psychol. Sci. 26, 559-569. doi: 10.1177/0956797614567341
94. Simonsohn, U., Nelson, L. D., and Simmons, J. P. (2014a). P-curve and effect size: correcting for publication bias using only significant results. Perspect. Psychol. Sci. 9, 666-681. doi: 10.1177/1745691614553988
95. Simonsohn, U., Nelson, L. D., and Simmons, J. P. (2014b). P-curve: a key to the file-drawer. J. Exp. Psychol. Gen. 143, 534-547. doi: 10.1037/a0033242
96. Spearman, C. (1904). '‘General intelligence’' objectively determined and measured. Am. J. Psychol. 15, 201-293. doi: 10.2307/1412107
97. Spence, I., Yu, J. J., Feng, J., and Marshman, J. (2009). Women match men when learning a spatial skill. J. Exp. Psychol. Learn. Mem. Cogn. 35, 1097-1103. doi: 10.1037/a0015641
98. Stankov, L., and Chen, K. (1988a). Can we boost fluid and crystallised intelligence? A structural modelling approach. Aust. J. Psychol. 40, 363-376. doi: 10.1080/00049538808260056
99. Stankov, L., and Chen, K. (1988b). Training and changes in fluid and crystallized intelligence. Contemp. Educ. Psychol. 13, 382-397. doi: 10.1016/0361-476x(88)90037-9
100. Stevens, C., Harn, B., Chard, D. J., Currin, J., Parisi, D., and Neville, H. (2013). Examining the role of attention and instruction in at-risk kindergarteners: electrophysiological measures of selective auditory attention before and after an early literacy intervention. J. Learn. Disabil. 46, 73-86. doi: 10.1177/0022219411417877
101. te Nijenhuis, J., van Vianen, A. E. M., and van der Flier, H. (2007). Score gains on g-loaded tests: no g. Intelligence 35, 283-300. doi: 10.1016/j.intell.2006.07.006
102. Thompson, T. W., Waskom, M. L., Garel, K.-L. A., Cardenas-Iniguez, C., Reynolds, G. O., Winter, R., et al. (2013). Failure of working memory training to enhance cognition or intelligence. PLoS One 8: e63614. doi: 10.1371/journal.pone.0063614
103. Tidwell, J. W., Dougherty, M. R., Chrabaszcz, J. R., Thomas, R. P., and Mendoza, J. L. (2014). What counts as evidence for working memory training? Problems with correlated gains and dichotomization. Psychon. Bull. Rev. 21, 620-628. doi: 10.3758/s13423-013-0560-7
104. Tippmann, S. (2015). Programming tools: adventures with R. Nature 517, 109-110. doi: 10.1038/517109a
105. Trafimow, D., and Marks, M. (2015). Editorial. Basic Appl. Soc. Psych. 37, 1-2. doi: 10.1080/01973533.2015.1012991
106. van Assen, M. A. L. M., van Aert, R. C. M., Nuijten, M. B., and Wicherts, J. M. (2014). Why publishing everything is more effective than selective publishing of statistically significant results. PLoS One 9: e84896. doi: 10.1371/journal.pone.0084896
107. Venables, W. N., and Ripley, B. D. (2002). Modern Applied Statistics with S. 4th Edn. New York, NY: Springer.
108. Vul, E., Harris, C., Winkielman, P., and Pashler, H. (2009). Puzzlingly high correlations in fmri studies of emotion, personality and social cognition. Perspect. Psychol. Sci. 4, 274-290. doi: 10.1111/j.1745-6924.2009.01125.x
109. Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychon. Bull. Rev. 14, 779-804. doi: 10.3758/bf03194105
110. Wagenmakers, E.-J., Verhagen, J., Ly, A., Bakker, M., Lee, M. D., Matzke, D., et al. (2015). A power fallacy. Behav. Res. Methods 47, 913-917. doi: 10.3758/s13428-014-0517-4
111. Wetzels, R., Grasman, R. P. P. P., and Wagenmakers, E.-J. (2012). A default Bayesian hypothesis test for anova designs. Am. Stat. 66, 104-111. doi: 10.1080/00031305.2012.695956
112. Wetzels, R., and Wagenmakers, E.-J. (2012). A default Bayesian hypothesis test for correlations and partial correlations. Psychon. Bull. Rev. 19, 1057-1064. doi: 10.3758/s13423-012-0295-x
113. Wickham, H. (2009). ggplot2: Elegant Graphics for Data Analysis. New York, NY: Springer.
114. Wickham, H. (2011). The split-apply-combine strategy for data analysis. J. Stat. Softw. 40, 1-29. doi: 10.18637/jss.v040.i01
115. Zelinski, E. M., Peters, K. D., Hindin, S., Petway, K. T., and Kennison, R. F. (2014). Evaluating the relationship between change in performance on training tasks and on untrained outcomes. Front. Hum. Neurosci. 8: 617. doi: 10.3389/fnhum.2014.00617
116. Zinke, K., Zeintl, M., Rose, N. S., Putzmann, J., Pydde, A., and Kliegel, M. (2014). Working memory training and transfer in older adults: effects of age, baseline performance and training gains. Dev. Psychol. 50, 304-315. doi: 10.1037/a0032982