Improving the Replicability of Psychological Science Through Pedagogy

Advances in Methods and Practices in Psychological Science

Volumn 1, Issue 1, 2018, Pages 7-18

  Hawkins, Robert X D ^a   Smith, Eric N ^a   Au, Carolyn ^a   Arias, Juan Miguel ^a   Catapano, Rhia ^a   Hermann, Eric ^a   Keil, Martin ^a   Lampinen, Andrew ^a   Raposo, Sarah ^a   Reynolds, Jesse ^a   Salehi, Shima ^a   Salloum, Justin ^a   Tan, Jed ^a   Frank, Michael C ^a  

^a STANFORD UNIVERSITY   (United States)

Author keywords

experimental methods; open data; open materials; pedagogy; preregistered; replication; reproducibility

Indexed keywords

EID: 85060208907     PISSN: 25152459     EISSN: 25152467     Source Type: Journal    
DOI: 10.1177/2515245917740427     Document Type: Article

References (47)

1. Anderson C. J., Bahnik Š., Barnett-Cowan M., Bosco F. A., Chandler J., Chartier C. R... Zuni K., (2016). Response to comment on “Estimating the reproducibility of psychological science.” Science, 351, 1037.
2. Atir S., Rosenzweig E., Dunning D., (2015). When knowledge knows no bounds: Self-perceived expertise predicts claims of impossible knowledge. Psychological Science, 26, 1295–1303.
3. Bakeman R., (2005). Recommended effect size statistics for repeated measures designs. Behavior Research Methods, 37, 379–384.
4. Buhrmester M., Kwang T., Gosling S. D., (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5.
5. Button K. S., Ioannidis J. P. A., Mokrysz C., Nosek B. A., Flint J., Robinson E. S. J., Munafò M. R., (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376.
6. Chandler J., Mueller P., Paolacci G., (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130.
7. Chandler J., Paolacci G., Peer E., Mueller P., Ratliff K. A., (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26, 1131–1139.
8. Crump M. J. C., McDonnell J. V., Gureckis T. M., (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLOS ONE, 8(3), Article e57410. doi:10.1371/journal.pone.0057410
9. Etz A., Vandekerckhove J., (2016). A Bayesian perspective on the Reproducibility Project: Psychology. PLOS ONE, 11(2), Article e0149794. doi:10.1371/journal.pone.0149794
10. Everett J. A. C., Earp B. D., (2015). A tragedy of the (academic) commons: Interpreting the replication crisis in psychology as a social dilemma for early-career researchers. Frontiers in Psychology, 6, Article 01152. doi:10.3389/fpsyg.2015.01152
11. Frank M. C., Saxe R., (2012). Teaching replication. Per-spectives on Psychological Science, 7, 600–604.
12. Gilbert D. T., King G., Pettigrew S., Wilson T. D., (2016). Comment on “Estimating the reproducibility of psychological science.” Science, 351, 1037.
13. Grahe J. E., Reifman A., Hermann A. D., Walker M., Oleson K. C., Nario-Redmond M., Wiebe R. P., (2012). Harnessing the undiscovered resource of student research projects. Perspectives on Psychological Science, 7, 605–607.
14. Hoenig J. M., Heisey D. M., (2001). The abuse of power. The American Statistician, 55, 1–6.
15. Inbar Y., (2016). Association between contextual dependence and replicability in psychology may be spurious. Proceedings of the National Academy of Sciences, USA, 113, E4933–E4934.
16. Ioannidis J. P. A., (2005). Why most published research findings are false. PLOS Medicine, 2(8), Article e124. doi:10.1371/journal.pmed.0020124
17. King M., Dablander F., Jakob L., Agan M. L. F., Huber F., Haslbeck J. M. B., Brecht K. F., (2016). Registered reports for student research. Journal of European Psychology Students, 7, 20–23.
18. Ko S. J., Sadler M. S., Galinsky A. D., (2015). The sound of power: Conveying and detecting hierarchical rank through voice. Psychological Science, 26, 3–14.
19. Koole S. L., Lakens D., (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7, 608–614.
20. Lakens D., (2013). Using a smartphone to measure heart rate changes during relived happiness and anger. IEEE Transactions on Affective Computing, 4, 238–241.
21. LeBel E., (2015). A new replication norm for psychology. Collabra, 1, Article 4. doi:10.1525/collabra.23
22. Lewis M. L., Frank M. C., (2016). Understanding the effect of social context on learning: A replication of Xu and Tenenbaum (2007b). Journal of Experimental Psychology: General, 145, e72–e80.
23. Lewis N. A., Oyserman D., (2015). When does the future begin? Time metrics matter, connecting present and fu-ture selves. Psychological Science 26, 816–825.
24. Lipsey M. W., Wilson D. B., (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
25. Liverence B. M., Scholl B. J., (2015). Object persistence enhances spatial navigation: A case study in smartphone vision science. Psychological Science, 26, 955–963.
26. Makel M. C., Plucker J. A., Hegarty B., (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7, 537–542.
27. Monin B., (2016). Be careful what you wish for: Commentary on Ebersole et al. (2016). Journal of Experimental Social Psychology, 67, 95–96.
28. Morey R. D., Hoekstra R., Rouder J. N., Lee M. D., Wagenmakers E.-J., (2016). The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 23, 103–123.
29. Nosek B. A., Alter G., Banks G. C., Borsboom D., Bowman S. D., Breckler S. J... Yarkoni T., (2015). Promoting an open research culture. Science, 348, 1422–1425.
30. Nuijten M. B., Hartgerink C. H. J., van Assen M. A. L. M., Epskamp S., Wicherts J. M., (2016). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 48, 1205–1226.
31. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716. doi:10.1126/science.aac4716
32. Phillips J., Ong D. C., Surtees A. D. R., Xin Y., Williams S., Saxe R., Frank M. C., (2015). A second look at automatic theory of mind: Reconsidering Kovács, Téglás, and Endress (2010). Psychological Science, 26, 1353–1367.
33. Proudfoot D., Kay A. C., Koval C. Z., (2015). A gender bias in the attribution of creativity: Archival and experimental evidence for the perceived association between masculinity and creative thinking. Psychological Science, 26, 1751–1761.
34. Rouder J. N., Speckman P. L., Sun D., Morey R. D., Iverson G., (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237.
35. Salehi N., Irani L. C., Bernstein M. S., Alkhatib A., Ogbe E., Milland K., Clickhappier. (2015). We are dynamo: Overcoming stalling and friction in collective action for crowd workers. In Begole B., (Ed.), Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 1621–1630). New York, NY: ACM.
36. Scopelliti I., Loewenstein G., Vosgerau J., (2015). You call it “self-exuberance”; I call it “bragging”: Miscalibrated predictions of emotional responses to self-promotion. Psychological Science, 26, 903–914.
37. Simons D. J., Shoda Y., Lindsay D. S., (2016). Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, 12, 1123–1128.
38. Simonsohn U., (2015). Small telescopes: Detectability and the evaluation of replication results. Psychological Science, 26, 559–569.
39. Sofer C., Dotsch R., Wigboldus D., Todorov A., (2015). What is typical is good: The influence of face typicality on perceived trustworthiness. Psychological Science, 26, 39–47.
40. Standing L. G., (2016). How to use replication team projects in a research methods course. Essays From X-Cellence in Teaching, XV, 26–31.
41. Stewart N., Ungemach C., Harris A. J. L., Bartels D. M., Newell B. R., Paolacci G., Chandler J., (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10, 479–491.
42. Storm B. C., Stone S. M., (2015). Saving-enhanced memory: The benefits of saving on the learning and remembering of new information. Psychological Science, 26, 182–188.
43. Van Bavel J. J., Mende-Siedlecki P., Brady W. J., Reinero D. A. (2016). Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences, USA, 113, 6454–6459.
44. Wagenmakers E.-J., Wetzels R., Borsboom D., van der Maas H. L., Kievit R. A., (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638.
45. Wang Z., Lukowski S. L., Hart S. A., Lyons I. M., Thompson L. A., Kovas Y... Petrill S. A., (2015). Is math anxiety always bad for math learning? The role of math motivation. Psychological Science, 26, 1863–1876.
46. Xu Y., Franconeri S. L., (2015). Capacity for visual features in mental rotation. Psychological Science, 26, 1241–1251.
47. Zaval L., Markowitz E. M., Weber E. U., (2015). How will I be remembered? Conserving the environment for the sake of one’s legacy. Psychological Science, 26, 231–236.