A call for examining replication and bias in special education research

Remedial and Special Education

Volumn 35, Issue 4, 2014, Pages 233-246

  Cook, Bryan G ^a  

^a UNIVERSITY OF HAWAII   (United States)

Author keywords

Evidence based practice; Instructional evaluation; Quantitative; Research methodology

Indexed keywords

EID: 84927658409     PISSN: 07419325     EISSN: 15384756     Source Type: Journal    
DOI: 10.1177/0741932514528995     Document Type: Article

References (102)

1. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543-554. doi:10.1177/1745691612459060
2. Bakker, M., & Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research, 43, 666-678. doi:10.3758/s13428-011-0089-5
3. Barchard, K. A., & Pace, L. A. (2011). Preventing human error: The impact of data entry methods on data accuracy and statistical results. Computers in Human Behavior, 27, 1834-1839. doi:10.1016/j.chb.2011.04.004
4. Begley, C. G., & Ellis, E. M. (2012). Raise standards for preclinical cancer research. Nature, 483, 531-533. doi:10.1038/483531a
5. Begley, S. (2012, August 14). More trial, less error: An effort to improve scientific studies. Reuters. Retrieved from http://www.reuters.com/article/2012/08/14/us-science-replicationservice-idUSBRE87D0FV20120814
6. Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407-425. doi:10.1037/a0021524
7. Bergstrom, J. P., Partington, S., Murphy, M. K., Galvao, L., Fayram, E., & Cisler, R. A. (2009). Active consent in urban elementary schools: An examination of demographic differences in consent rates. Evaluation Review, 33, 481-496. doi:10.1177/0193841X09339987
8. Berliner, D. C. (2002). Educational research: The hardest science of all. Educational Researcher, 31(8), 18-20. doi:10.3102/0013189X031008018
9. Boardman, A. G., Arguelles, M. E., Vaughn, S., Hughes, M. T., & Klingner, J. (2005). Special education teachers’ views of research-based practices. Journal of Special Education, 39, 168-180.
10. Bornmann, L., Mutz, R., & Daniel, H. D. (2010). A reliabilitygeneralization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PloS ONE, 5(12), e14331. doi:10.1371/journal.pone.0014331
11. Boutron, I., Dutton, S., Ravaud, P., & Altman, D. G. (2010). Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. Journal of American Medical Association, 303, 2058-2064. doi:10.1001/jama.2010.651
12. Casadevall, A., & Fang, F. C. (2012). Reforming science: Methodological and cultural reforms. Infection and Immunity, 80, 891-896. doi:10.1128/IAI.06183-11
13. Chapman, S., Ragg, M., & McGeehan, K. (2009). Citation bias in reported smoking prevalence in people with schizophrenia. Australian and New Zealand Journal of Psychiatry, 43, 277-282.
14. Cook, B. G., & Odom, S. L. (2013). Evidence-based practices and implementation science in special education. Exceptional Children, 79, 135-144.
15. Cook, B. G., Smith, G. J., & Tankersley, M. (2012). Evidence-based practices in education. In K. R. Harris, S. Graham, & T. Urdan (Eds.), APA educational psychology handbook (Vol. 1, pp. 495-528). Washington, DC: American Psychological Association.
16. Cook, B. G., Tankersley, M., & Landrum, T. J. (2009). Determining evidence-based practices in special education. Exceptional Children, 75, 365-383.
17. Council for Exceptional Children. (2014). Council for Exceptional Children standards for evidence-based practices in special education. Retrieved from http://www.cec.sped.org/Standards/Evidence-Based-Practice-Resources-Original
18. Courser, M. W., Shamblen, S. R., Lavrakas, P. J., Collins, D., & Ditterline, P. (2009). The impact of active consent procedures on nonresponse and nonresponse error in youth survey data: Evidence from a new experiment. Evaluation Review, 33, 370-395. doi:10.1177/0193841X09337228
19. Dammann, J. E., & Vaughn, S. (2001). Science and sanity in special education. Behavioral Disorders, 27, 21-29.
20. Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A., Cronin, E., Williamson, P. R. (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE, 3(8), e3081. doi:10.1371/journal.pone.0003081
21. Egger, M., Smith, G. D., & Sterne, J. A. (2001). Uses and abuses of meta-analysis. Clinical Medicine, 1, 478-484. doi:10.7861/clinmedicine.1-6-478
22. Evanschitzky, H., Baumgarth, C., Hubbard, R., & Armstrong, J. S. (2007). Replication research’s disturbing trend. Journal of Business Research, 60, 411-415. doi:10.1016/j.jbusres.2006.12.003
23. Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(5), e5738. doi:10.1371/journal.pone.0005738
24. Fang, F. C., & Casadevall, A. (2012). Reforming science: Structural reforms. Infection and Immunity, 80, 897-901. doi:10.1128/IAI.06184-11
25. Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120-128. doi:10.1037/a0024445
26. Flaherty, D. K. (2011). The vaccine-autism connection: A public health crisis caused by unethical medical practices and fraudulent science. Annals of Pharmacotherapy, 45, 1302-1304. doi:10.1345/aph.1Q318
27. Francis, G. (2012a). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7, 585-594. doi:10.1177/1745691612459520
28. Francis, G. (2012b). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19, 151-156. doi:10.3758/s13423-012-0227-9
29. Freed, G. L., Clark, S. J., Butchart, A. T., Singer, D. C., & Davis, M. M. (2011). Parental vaccine safety concerns in 2009. Pediatrics, 125, 654-659. doi:10.1542/peds.2009-1962
30. Freeman, D. H. (2010). Lies, damned lies, and medical science. The Atlantic. Retrieved from http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-andmedical-science/8269
31. Galak, J., LeBoeuf, R. A., Nelson, L. D., & Simmons, J. P. (2012). Correcting the past: Failures to replicate psi. Journal of Personality and Social Psychology, 103, 933-948. doi:10.1037/a0029709
32. Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood, C., & Innocenti, M. S. (2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71, 149-164.
33. Giner-Sorolla, R (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7, 562-571.
34. Haynes, A. B., Weiser, T. G., Berry, W. R., Lipsitz, S. R., Breizat, A. H. S., Dellinger, E. P., & Gawande, A. A. (2009). A surgical safety checklist to reduce morbidity and mortality in a global population. New England Journal of Medicine, 360, 491-499. doi:10.1056/NEJMsa0810119
35. Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009). Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database of Systematic Reviews, Article MR000006. doi:10.1002/14651858.MR000006.pub3
36. Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practices in special education. Exceptional Children, 71, 165-171.
37. Hrobjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I., & Brorson, S. (2012). Observer bias in randomised clinical trials with binary outcomes: Systematic review of trials with both blinded and non-blinded outcome assessors. British Medical Journal, 344, Article e1119. doi:10.1136/bmj.e1119
38. Ioannidis, J. P. A. (2005a). Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association, 294, 218-228. doi:10.1001/jama.294.2.218
39. Ioannidis, J. P. A. (2005b). Why most published research findings are false. PLoS Medicine, 2(8), e124. doi:10.1371/journal.pmed.0020124
40. Ioannidis, J. P. A. (2006). Evolution and translation of research findings: From bench to where. PLoS Hub for Clinical Trials, 1(7), e36. doi:10.1371/journal.pctr.0010036
41. Ioannidis, J. P. A. (2008). Perfect study, poor evidence: Interpretation of biases preceding study design. Seminars in Hematology, 45, 160-166.
42. Ioannidis, J. P. A. (2010). Meta-research: The art of getting it wrong. Research Synthesis Methods, 1, 169-184. doi:10.1002/jrsm.19
43. Ioannidis, J. P. A. (2012a). Scientific communication is down at the moment, please check again later. Psychological Inquiry, 23, 267-270. doi:10.1080/1047840X.2012.699427
44. Ioannidis, J. P. A. (2012b). Why science is not necessarily selfcorrecting. Perspectives on Psychological Science, 7, 645-654. doi:10.1177/1745691612464056
45. Ioannidis, J. P. A., Haidich, A. B., Pappa, M., Pantazis, N., Kokori, S. I., Tektonidou, M. G., & Lau, J. (2001). Comparison of evidence of treatment effects in randomized and nonrandomized studies. Journal of the American Medical Association, 286, 821-830. doi:10.1001/jama.286.7.821
46. Iyer, R., & Graham, J. (2012). Leveraging the wisdom of crowds in a data-rich utopia. Psychological Inquiry, 23, 271-273. doi :10.1080/1047840X.2012.705244
47. Jannot, A. S., Agoritsas, T., Gayet-Ageron, A., & Perneger, T. V. (2013). Citation bias favoring statistically significant studies was present in medical research. Journal of Clinical Epidemiology, 66, 296-301. http://dx.doi.org/10.1016/j. jclinepi.2012.09.015
48. Jasny, B. R., Chin, G., Chong, L., & Vignieri, S. (2011). Again, and again, and again . . . Science, 334, 1225. doi:10.1126/science.334.6060.1225
49. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524-532. doi:10.1177/0956797611430953
50. Kauffman, J. M. (1999). How we prevent the prevention of emotional and behavioral disorders. Exceptional Children, 65, 448-468.
51. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196-217. doi:10.1207/s15327957pspr0203_4
52. Kocakaya, S. (2011). An educational dilemma: Are educational experiments working? Educational Research and Reviews, 6, 110-123.
53. Kratochwill, T. R., Hitchcock, J. H., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., & Shadish, W. R. (2013). Single-case intervention research design standards. Remedial and Special Education, 34, 26-38. doi:10.1177/0741932512452794
54. Kriegeskorte, N. (2012). Open evaluation: A vision for entirely transparent post-publication peer review and rating for science. Frontiers in Computational Neuroscience, 6(79), 1-18. doi:10.3389/fncom.2012.00079
55. Kunz, R., Vist, G. E., & Oxman, A. D. (2008). Randomisation to protect against selection bias in healthcare trials. Cochrane Database of Systematic Reviews, 2007, Article MR000012. doi:10.1002/14651858.MR000012.pub2
56. Lambert, J. (2011). Statistics in brief: How to assess bias in clinical studies. Clinical Orthopaedics and Related Research, 469, 1794-1796. doi:10.1007/s11999-010-1538-7
57. Lehrer, J. (2010). The truth wears off: Is there something wrong with the scientific method? The New Yorker. Retrieved from http://www.newyorker.com/reporting/2010/12/13/101213fa_ fact_lehrer
58. LeLorier, J., Gregoire, G., Benhaddad, A., Lapierre, J., & Derderian, F. (1997). Discrepancies between meta-analyses and subsequent large randomized, controlled trials. New England Journal of Medicine, 337, 536-542. doi:10.1056/NEJM199708213370806
59. Lipsey, M. W., Puzio, K., Yun, C., Hebert, M. A., Steinka-Fry, K., Cole, M. W., Busick, M. D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms (NCSER 2013-3000). Washington, DC: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncser/pubs/20133000/
60. Makel, M., Plucker, J., & Hegarty, B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7, 537-542. doi:10.1177/1745691612460688
61. Masicampo, E. J., & Lalande, D. R. (2012). A peculiar presence of p values just below .05. The Quarterly Journal of Experimental Psychology, 65, 2271-2279. doi:10.1080/17470218.2012.711335
62. Mathieu, S., Boutron, I., Moher, D., Altman, D. G., & Ravaud, P. (2009). Comparison of registered and published primary outcomes in randomized controlled trials. Journal of the American Medical Association, 302, 977-984. doi:10.1001/jama.2009.1242
63. Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147-163. doi:10.1037/1082-989X.9.2.147
64. Moonesinghe, R., Khoury, M. J., & Janssens, A. C. J. (2007). Most published research findings are false—But a little replication goes a long way. PLoS Medicine, 4(2), e28. doi:10.1371/journal.pmed.0040028
65. Munafo, M. R., & Flint, J. (2010). How reliable are scientific studies. British Journal of Psychiatry, 197, 257-258. doi:10.1192/bjp.bp.109.069849
66. Naik, G. (2011, December 2). Scientists’ elusive goal: Reproducing study results. The Wall Street Journal. Retrieved from http://online.wsj.com/article/SB1000142405297020376480457705 9841672541590.html
67. Nieminen, P., Rucker, G., Miettunen, J., Carpenter, J., & Schumacher, M. (2007). Statistically significant papers in psychiatry were cited more often than others. Journal of Clinical Epidemiology, 60, 939-946. doi:10.1016/j.clinepi.2006.11.014
68. Nosek, B. A., & Bar-Anan, Y. (2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23, 217-243. doi:10.1080/1047840X.2012.692215
69. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia II: Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631. doi:10.1177/1745691612459058
70. Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531-536. doi:10.1177/1745691612463401
71. Pereira, T. V., & Ioannidis, J. P. A. (2011). Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. Journal of Clinical Epidemiology, 64, 1060-1069. doi:10.1016/j.jclinepi.2010.12.012
72. Peters, D., & Ceci, S. (1982). Peer-review practices of psychological journals: The fate of submitted articles, submitted again. Behavioral and Brain Sciences, 5, 187-255. doi:10.1017/S0140525X00011183
73. Prinz, F., Schlange, T., & Asadullah, K. (2011). Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews: Drug Discovery, 10, 712-713. doi:10.1038/nrd3439-c1PublicLibraryofScienceONE.(n.d.).PLOSONEjournalinformation.Retrievedfromhttp://www.plosone.org/static/information
74. Ridley, J., Kolm, N., Freckelton, R. P., & Gage, M. J. G. (2007). An unexpected influence of widely used significance thresholds on the distribution of reported p-values. Journal of Evolutionary Biology, 20, 1082-1089. doi:10.1111/j.1420-9101.2006.01291.x
75. Sackett, D. L. (1979). Bias in analytic research. Journal of Chronic Diseases, 32(1-2), 51-63.
76. San Francisco Declaration on Research Assessment. (2013). San Francisco declaration on research assessment: Putting science into the assessment of research. Retrieved from http://am.ascb.org/dora/files/SFDeclarationFINAL.pdf
77. Santoro, L., Gersten, R., & Newman-Gonchar, R. A. (2011). Designing rigorous group studies in special education. In J. M. Kauffman & D. P. Hallahan (Eds.), Handbook of special education (pp. 37-46). New York, NY: Routledge.
78. Schmidt, L. M., & Gotzsche, P. C. (2005). Of mites and men: Reference bias in narrative reviews: A systematic review. Journal of Family Practice, 54, 334-338.
79. Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13, 90-100. doi:10.1037/a0015108
80. Schooler, J. (2011). Unpublished results hide the decline effect. Nature, 470, 437.
81. Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., & Smith, R. (2008). What errors do peer reviewers detect, and does training improve their ability to detect them? Journal of the Royal Society of Medicine, 101, 507-514. doi:10.1258/jrsm.2008.080062
82. Schulz, K. F., Chalmers, I., Hayes, R. J., & Altman, D. G. (1995). Empirical evidence of bias. Journal of the American Medical Association, 273, 408-412. doi:10.1001/jama.273.5.408
83. Seethaler, P. M., & Fuchs, L. S. (2005). A drop in the bucket: Randomized controlled trials testing reading and math interventions. Learning Disabilities Research & Practice, 20, 98-102. doi:10.1111/j.1540-5826.2005.00125.x
84. Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments. Journal of the American Statistical Association, 103, 1334-1343. doi:10.1198/01621450800000073
85. Simmerman, S., & Swanson, H. L. (2001). Treatment outcomes for students with learning disabilities: How important are internal and external validity? Journal of Learning Disabilities, 34, 221-236. doi:10.1177/002221940103400303
86. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. doi:10.1177/0956797611417632
87. Slavin, R. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher, 31(7), 15-21. doi:10.3102/0013189X031007015
88. Song, F., Parekh-Bhurke, S., Hooper, L., Loke, Y. K., Ryder, J. J., Sutton, J. J., Harvey, I. (2009). Extent of publication bias in different categories of research cohorts: A meta-analysis of empirical studies. BMC Medical Research Methodology, 9, Article 79. doi:10.1186/1471-2288-9-79
89. Song, M., & Herman, R. (2010). Critical issues and common pitfalls in designing and conducting impact studies in education: Lessons learned From the What Works Clearinghouse (Phase I). Educational Evaluation and Policy Analysis, 32, 351-371. doi:10.3102/0162373710373389
90. Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association, 54, 30-34. http://www.jstor.org/stable/2282137
91. Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49, 108-112. http://www.jstor.org/stable/2684823
92. Tatsioni, A., Bonitsis, N. G., & Ioannidis, J. P. A. (2007). Persistence of contradicted claims in the literature. Journal of the American Medical Association, 298, 2517-2526. doi:10.1001/jama.298.21.2517
93. Tenopir, C., Allard, S., Douglass, K., Aydinoglu, A. U., Wu, L., Read, E., Frame, M. (2011). Data sharing by scientists: Practices and perceptions. PLoS ONE, 6(6), e21101. doi:10.1371/journal.pone.0021101
94. Torgerson, D. J., & Torgerson, C. J. (2003). Avoiding bias in randomised controlled trials in educational research. The British Journal of Educational Studies, 51, 36-45. http://www.jstor.org/stable/3121643
95. Unreliable research: Trouble at the lab. (2013, October 19). The Economist. Retrieved from http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correctingalarming-degree-it-not-trouble
96. Van Noorden, R. (2011). The trouble with retractions. Nature, 478, 26-28. doi:10.1038/478026a
97. Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632-638. doi:10.1177/1745691612463078
98. Wakefield, A. J., Murch, S. H., Anthony, A., Linnell, J., Casson, D. M., Malik, M. M., . . .Walker-Smith, J. A. (1998). Ileal lymphoid nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Lancet, 351, 637-641. doi:10.1016/S0140-6736(97)11096-0
99. Wicherts, J. M., Kievit, R. A., Bakker, M., & Borsboom, D. (2012). Letting the daylight in: Reviewing the reviewers and other ways to maximize transparency in science. Frontiers in Computational Neuroscience, 6, 20. doi:10.3389/fncom.2012.00020
100. Yarkoni, T. (2012). Beginning at Nosek and Bar-Anan’s end: Let’s put open evaluation first. Psychological Inquiry, 23, 305-307. doi:10.1080/1047840X.2012.706204
101. Yavchitz, A., Boutron, I., Bafeta, A., Marroun, I., Charles, P., Mantz, J., & Ravaud, P. (2012). Misrepresentation of randomized controlled trials in press releases and news coverage: A cohort study. PLoS Medicine, 9(9), e1001308. doi:10.1371/journal.pmed.1001308
102. Yong, E. (2012). Replication studies: Bad copy. Nature, 485, 298-300. doi:10.1038/485298a