Facts Are More Important Than Novelty: Replication in the Education Sciences

Educational Researcher

Volumn 43, Issue 6, 2014, Pages 304-316

  Makel, Matthew C ^a   Plucker, Jonathan A ^b  

^a DUKE UNIVERSITY   (United States)

^b UNIVERSITY OF CONNECTICUT   (United States)

Author keywords

assessment; content analysis; educational policy; evaluation; replication; research methodology

Indexed keywords

EID: 84906731467     PISSN: 0013189X     EISSN: 1935102X     Source Type: Journal    
DOI: 10.3102/0013189X14545513     Document Type: Article

References (125)

1. Agresti A.An introduction to categorical data analysis. 2007). An introduction to categorical data analysis (2nd ed.). New York, NY: Wiley.New York, NY: Wiley; 2007:.
2. Ahlgren A.A modest proposal for encouraging replication.American Psychologist. 1969;24:471.
3. AndersonR. G.GreenW. H.McCulloughB. D.VinodH. D. (2005). The role of data and program code archives in the future of economic research. Working paper, Federal Reserve Bank of St. Louis, MO. Retrieved from http://research.stlouisfed.org/wp/2005/2005-014.pdf.
4. Bakan D.Test of significance in psychological research.Psychological Bulletin. 1966;66:423-437.
5. Bakker M.,van Dijk A.,Wicherts J. M.The rules of the game called psychological science.Perspectives in Psychological Science. 2012;7:543-554.
6. Bakker M.,Wicherts J. M.The (mis)reporting of statistical results in psychology journals.Behavior Research Methods. 2011;43:666-678.
7. Begley C. G.,Ellis L. M.Raise standards for preclinical cancer research.Nature. 2012;483:531-533.
8. Bem D. J.Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect.Journal of Personality and Social Psychology. 2011;100:407-425.
9. Bissell M.The risks of the replication drive.Nature. 2013;503:333-334.
10. Bozarth J. D.,Roberts R. R.Signifying significant significance.American Psychologist. 1972;27:774-775.
11. Brandt M. J.,IJzerman H.,Dijksterhuis A.,Farach F. J.,Geller J.,Giner-Sorolla R.,.van't Veer A.The replication recipe: What makes for a convincing replication?.Journal of Experimental Social Psychology. 2014;50:217-224.
12. Campbell D.,Stanley J.Experimental and quasi-experimental designs for research. Chicago, IL: Rand-McNally; 1963:.
13. Carpenter S.Psychology's bold initiative.Science. 2012;335:1558-1561.
14. Cesario J.Priming, replication, and the hardest science.Perspectives on Psychological Science. 2014;9:40-48.
15. Cicchetti D. V.The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation.Behavioral Brain Science. 1991;14:119-186.
16. Cohen J.The Earth is round (p <.05).American Psychologist. 1994;49:997-1003.
17. Cole S.Making science: Between nature and society. Cambridge, MA: Harvard University Press; 1992:.
18. Collins H. M.Changing order. London, UK: Sage; 1985:.
19. Crocker J.,Cooper M. L.Addressing scientific fraud.Science. 2011;334 (6060): 1182.
20. Cumming G.The new statistics: Why and how.Perspectives on Psychological Science. 2014;25:7-29.
21. de Winter J.,Happee R.Why Selective publication of statistically significant results can be effective.PLoS One. 2013;8:e66463.
22. Dewald W. G.,Thursby J. G.,Anderson R. G.Replication in empirical economics: The journal-of-money-credit-and-banking project.American Economic Review. 1986;76:587-603.
23. Eden D.Replication, meta-analysis, scientific progress, and AMJ's publication policy.Academy of Management Journal. 2002;45:841-846.
24. Retraction: Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children.The Lancet. 2010;375 (9713): 445.
25. Eich R.Business not as usual.Psychological Science. 2014;25:3-6.
26. Evanschitzky H.,Baumgarth C.,Hubbard R.,Armstrong J. S.Replication research's disturbing trend.Journal of Business Research. 2007;60:411-415.
27. Fanelli D."Positive" results increase down the hierarchy of the sciences.PLoS One. 2010;5:.
28. Fanelli D.Negative results are disappearing from most disciplines and countries.Scientometrics. 2012;90:891-904.
29. Frank M. C.,Saxe R.Teaching replication.Perspectives in Psychological Science. 2012;7:600-604.
30. Fryer R. G.Financial incentives and student achievement: Evidence from randomized trials.Quarterly Journal of Economics. 2011;126:1755-1798.
31. Fryer R. G.Teacher incentives and student achievement: Evidence from New York City public schools.Journal of Labor Economics. 2013;31:273-407.
32. Furchtgott E.Replicate, again and again.American Psychologist. 1984;39:1315-1316.
33. Golden M. A.Replication and non-quantitative research.PS: Political Science & Politics. 1995;28:481-483.
34. Grahe J. E.,Reifman A.,Hermann A. D.,Walker M.,Oleson K. C.,Nario-Redmond M.,Wiebe R. P.Harnessing the undiscovered resource of student research projects.Perspectives in Psychological Science. 2012;7:605-607.
35. Hansen W. B.Was Herodotus correct?.Prevention Science. 2011;12:118-120.
36. Hartshome J. K.,Schachner A.Tracking replicability as a method of post-publication open evaluation.Frontiers in Computational Neuroscience. 2012;6:1-14.
37. Hornig M.,Briese T.,Buie T.,Bauman M. L.,Lauwers G.,Siemetzki U.,.Lipkin W. I.Lack of association between measles virus vaccine and autism with enteropathy: A case-control study.PLoS One. 2008;3 (9): e3140.
38. Hubbard R.,Armstrong J. S.Replications and extensions in marketing: Rarely published but quite contrary.International Journal of Research in Marketing. 1994;11:233-248.
39. Hubbard R.,Vetter D. E.An empirical comparison of published replication research in accounting economics, finance, management, and marketing.Journal of Business Research. 1996;35:153-164.
40. Ioannidis J. P. A.Contradicted and initially stronger effects in highly cited clinical research.Journal of the American Medical Association. 2005a;294:218-228.
41. Ioannidis J. P. A.Contradictions in highly cited clinical research: Reply.Journal of the American Medical Association. 2005b;294:2696-2696.
42. Ioannidis J. P. A.Why most published research findings are false.PLoS Medicine. 2005c;2:696-701.
43. Ioannidis J. P. A.Why science is not necessarily self-correcting.Perspectives in Psychological Science. 2012;7:645-654.
44. Ioannidis J. P. A.,Khoury M. J.Improving validation practices in "omics" research.Science. 2011;334:1230-1232.
45. Jasny B. R.,Chin G.,Chong L.,Vignieri S.Introduction: Again, and again, and again...Science. 2011;334:1225-1225.
46. John L. K.,Loewenstein G.,Prelec D.Measuring the prevalence of questionable research practices with incentives for truth telling.Psychological Science. 2012;23:524-532.
47. Johnson V.Revised standard for statistical evidence.Proceedings of the National Academy of Science. 2013;110:19313-19317.
48. Jones K. S.,Derby P. L.,Schmidlin E. A.An investigation of the prevalence of replication research in human factors.Human Factors. 2010;52:586-595.
49. Kane E. J.Why journal editors should encourage the replication of applied econometric research.Quarterly Journal of Business and Economics. 1984;23:3-8.
50. Kelly C. W.,Chase L. J.,Tucker R. K.Replication in experimental communication research: An analysis.Human Communication Research. 1979;5:338-342.
51. Kerr N. L.HARKing: Hypothesizing after the results are known.Personality and Social Psychology Review. 1998;2:196-217.
52. Kerr S.,Tolliver J.,Petree D.Manuscript characteristics which influence acceptance for management and social-science journals.Academy of Management Journal. 1977;20:132-141.
53. King G.Replication, replication.Political Science & Politics. 1995;28:444-452.
54. Klein E. B.,Gould L. J.,Corey M.Social desirability in children: An extension and replication.Journal of Consulting and Clinical Psychology. 1969;33:128.
55. KleinR. A.RatliffK. A.VianelloM.AdamsR. B.BahnikS.BernsteinM. J. NosekB. A. (2013). Investigating variation in replicability: The "many labs" replication project. Retrieved from https://osf.io/wx7ck/files/ManyLabsManuscript.pdf/.
56. Kruskal W.,Majors R.Concepts of relative importance in recent scientific literature.American Statistician. 1989;43:2-6.
57. La Sorte M. A.Replication as a verification technique in survey research: A paradigm.Sociological Quarterly. 1972;13:218-277.
58. LeBel E. P.,Peters K. R.Fearing the future of empirical psychology: Bem's (2011) evidence of psi as a case study of deficiencies in modal research practice.Review of General Psychology. 2011;15:371-379.
59. LevittS. D.ListJ. A.NeckermannS.SadoffS. (2011). The impact of short-term incentives on student performance. Retrieved from http://bfi.uchicago.edu/events/20111028_experiments/papers/Levitt_List_Neckermann_Sadoff_Short-Term_Incentives_September2011.pdf.
60. Levy L. H.Reflections on replications and the experimenter bias effect.Journal of Consulting and Clinical Psychology. 1969;33:15-17.
61. Lewandowsky S.,Ecker U. K. H.,Seifert C. M.,Schwarz N.,Cook J.Misinformation and its correction: Continued influence and successful debiasing.Psychological Science in the Public Interest. 2012;13:106-131.
62. Lindsay R. M.,Ehrenberg A. S. C.The design of replicated studies.American Statistician. 1993;47:217-228.
63. Loevinger J.The "information explosion.".American Psychologist. 1968;23:455.
64. Lubin A.Replicability as a publication criterion.American Psychologist. 1957;12:519-520.
65. Lykken D. T.Statistical significance in psychological research.Psychological Bulletin. 1968;70:151-159.
66. Madden C. S.,Easley R. W.,Dunn M. G.How journal editors view replication research.Journal of Advertising. 1995;24:77-87.
67. Makel M. C.The empirical march: Making science better at self-correction.Psychology of Aesthetics, Creativity, and the Arts. 2014;8:2-7.
68. Makel M. C.,Plucker J. A.Creativity is more than novelty: Reconsidering replication as a creativity act.Psychology of Aesthetics, Creativity, and the Arts. 2014;8:27-29.
69. Makel M. C.,Plucker J. A.,Hegarty B.Replications in psychology research: How often do they really occur?.Perspectives in Psychological Science. 2012;7:537-542.
70. Maxwell S. E.The persistence of underpowered studies in psychological research: Causes, consequences, and remedies.Psychological Methods. 2004;9:147-163.
71. McBee M. T.,Matthews M. S.Change starts with journal editors: In response to Makel (2014).Psychology of Aesthetics, Creativity, and the Arts. 2014a;8:8-10.
72. McBee M. T.,Matthews M. S.Welcoming quality non-significance and replication work, but not the p-values: Announcing new policies for quantitative research.Journal of Advanced Academics. 2014b;25:68-78.
73. Meier K. J.Publishing replications: OK, let's try it.Political Science & Politics. 1995a;28:662-663.
74. National Reading Panel. (1999). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. Washington, DC: National Institute of Child Health and Human Development. Retrieved from https://www.nichd.nih.gov/publications/pubs/nrp/Documents/report.pdf.
75. Neuliep J. W.,Crandall R.Editorial bias against replication research.Journal of Social Behavior and Personality. 1990;5:85-90.
76. Neuliep J. W.,Crandall R.Everyone was wrong: There are lots of replications out there.Journal of Social Behavior and Personality. 1993a;8:1-8.
77. Neuliep J. W.,Crandall R.Reviewer bias against replication research.Journal of Social Behavior and Personality. 1993b;8:21-29.
78. Nosek B. A.,Spies J. R.,Motyl M.Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability.Perspectives in Psychological Science. 2012;7:615-631.
79. An open, large-scale collaborative effort to estimate the reproducibility of psychological science.Perspectives on Psychological Science. 2012;7:657-660.
80. Ostrom T. H.To replicate or explicate.American Psychologist. 1971;26:312.
81. Pashler H.,Harris C. R.Is the replicability crisis overblown? Three arguments examined.Perspectives in Psychological Science. 2012;7:531-536.
82. Pashler H.,Wagenmakers E. J.Introduction to the special section on replicability in psychological science: A crisis of confidence?.Perspectives on Psychological Science. 2012;7:528-530.
83. Peng R. D.Reproducible research in computational science.Science. 2011;334:1226-1227.
84. Perspectives on Psychological Science. (2012). Special section on replicability in psychological science: A crisis of confidence? Retrieved from http://pps.sagepub.com/content/7/6.toc.
85. Peters C. C.An example of replication of an experiment for increased reliability.Journal of Educational Research. 1938;32:3-9.
86. Peters D. P.,Ceci S. J.Peer-reviewed practices of psychological journals: The fate of accepted published articles, submitted again.Behavioral Brain Science. 1982;5:187-195.
87. Pigott T. D.,Valentine J. C.,Polanin J. R.,Williams R. T.,Canada D. D.Outcome-reporting bias in education research.Educational Researcher. 2013;:.
88. Plucker J.,Beghetto R. A.,Dow G.Why isn't creativity more important to educational psychologists? Potential, pitfalls, and future directions in creativity research.Educational Psychologist. 2004;39:83-96.
89. Powell K.Going against the grain.PLoS Biology. 2007;12:2748-2753.
90. Prinz F.,Schlange T.,Asadullah K.Believe it or not: How much can we rely on published data on potential drug targets?.Nature Reviews Drug Discovery. 2011;10:712-713.
91. Rhine J. B.Extra-sensory perception. Boston, MA: Branden; 1997:.
92. Ritchie S. J.,Wiseman R.,French C. C.Replication, replication, replication.Psychologist. 2012;25:346-348.
93. RoedigerH. L. (2012). Psychology's woes and a partial cure: The value of replication. Academic Observer. Retrieved from http://www.psychologicalscience.org/index.php/publications/observer/2012/february-12/psychologys-woes-and-a-partial-cure-the-value-of-replication.html.
94. Rosenthal R.Experimenter effects in behavioral research. East Norwalk, CT: Appleton-Century-Crofts; 1966:.
95. Rosenthal R.Covert communication in the psychological experiment.Psychological Bulletin. 1967;67:356-367.
96. Rosenthal R.On not so replicated experiments and not so null results.Journal of Consulting and Clinical Psychology. 1969;33:7-10.
97. Rosenthal R.The file drawer problem and tolerance for null results.Psychological Bulletin. 1979;86:638-641.
98. Rotton J.,Foos P. W.,Vanmeek L.,Levitt M.Publication practices and the file drawer problem: A survey of published authors.Journal of Social Behavior and Personality. 1995;10:1-13.
99. Rozeboom W. W.The fallacy of the null-hypothesis significance test.Psychological Bulletin. 1960;57:416-428.
100. Ryan M. J.Replication in field biology: The case of the frog-eating bat.Science. 2011;334:1229-1230.
101. Sagan C.The demon-haunted world: Science as a candle in the dark. New York, NY: Ballantine Books; 1997:.
102. Schimmack U.The ironic effect of significant results on the credibility of multiple-study articles.Psychological Methods. 2012;17:551-566.
103. Schlosberg H.Repeating fundamental experiments.American Psychologist. 1951;6:177-177.
104. Schmidt S.Shall we really do it again? The powerful concept of replication is neglected in the social sciences.Review of General Psychology. 2009;13:90-100.
105. SchoolerJ. (2011). Unpublished results hide the decline effect. Nature, 470, 437-437. doi:10.1038/470437a.
106. Simons D. J.The need for new incentives.Psychologist. 2012;25:349.
107. Simons D. J.,Holcombe A. O.Registered replication reports: A new article type at.Perspectives on Psychological Science. APS Observer. 2014;27 (3): 16-17.
108. Simmons J. P.,Nelson L. D.,Simonsohn U.False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.Psychological Science. 2011;22:1359-1366.
109. Spellman B. A.Introduction to the special section: Data, data, everywhere.especially in my file drawer.Perspectives on Psychological Science. 2012;7:58-59.
110. Sterling T. D.Publication decisions and their possible effects on inferences drawn from tests of significance-or vice versa.Journal of the American Statistical Association. 1959;54:30-34.
111. Sterling T. D.,Rosenbaum W. L.,Weinkam J. J.Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice-versa.American Statistician. 1995;49:108-112.
112. Tarnowski K. J.,Drabman R. S.,Anderson D. F.,Kelly P. A.Disproportionate referrals for child academic behavior problems: Replication and extension.Journal of Consulting and Clinical Psychology. 1990;58:240-243.
113. Taylor B.,Miller E.,Farrington C.,Petropoulos M. C.,Favot-Mayaud I.,Li J.,Waight P. A.Autism and measles, mumps, and rubella vaccine: No epidemiological evidence for a causal association.The Lancet. 1999;353 (9169): 2026-2029.
114. Tomasello M.,Call J.Methodological challenges in the study of primate cognition.Science. 2011;334:1227-1228.
115. Tukey J. W.Analyzing data: Sanctification or detective work?.American Psychologist. 1969;24:83-91.
116. Valentine J. C.,Biglan A.,Boruch R. F.,Castro F. G.,Collins L. M.,Flay B. R.,.Schinke S. P.Commentaries on "Replication in Prevention Science": A rejoinder.Prevention Science. 2011;12:123-125.
117. Van Noorden R.The trouble with retractions.Nature. 2011;478:26-28.
118. Wagenmakers E. J.,Wetzels R.,Borsboom D.,van der Mass H.Why psychologists must change the way they analyze their data: The case of psi.Journal of Personality and Social Psychology. 2011;100:426-432.
119. Wagenmakers E. J.,Wetzels R.,Borsboom D.,van der Maas H. L. J.,Kievit R. A.An agenda for purely confirmatory research.Perspectives in Psychological Science. 2012;7:632-638.
120. What Works Clearinghouse. (2013). Home page. Retrieved from http://ies.ed.gov/ncee/wwc/.
121. WhitehurstG. (2003). The Institute of Education Sciences: New wine, new bottles. A presentation by IES director Grover (Russ) Whitehurst. Retrieved from http://www2.ed.gov/rschstat/research/pubs/ies.html.
122. Williams S. N.Replication initiative: Prioritize publication.Science. 2012;336:801-802.
123. Winters B. A.But I don't have anything to write about.Library Acquisitions-Practice and Theory. 1996;20:391-394.
124. Yong E.Bad copy.Nature. 2012;485:298-300.
125. Yuan K.,Le V.,McCaffrey D. F.,Marsh J. A.,Hamilton L. S.,Stecher B. M.,Springer M. G.Incentive pay programs do not affect teacher motivation or reported practices: Results from three randomized studies.Educational Evaluation and Policy Analysis. 2013;35:3-22.