From old school to open science: The implications of new research norms for educational psychology and beyond

Educational Psychologist

Volumn 56, Issue 2, 2021, Pages 79-89

  Gehlbach, Hunter ^a   Robinson, Carly D ^b  

^a JOHNS HOPKINS UNIVERSITY   (United States)

^b BROWN UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85104052617     PISSN: 00461520     EISSN: None     Source Type: Journal    
DOI: 10.1080/00461520.2021.1898961     Document Type: Article

References (47)

1. Association for Psychological Science (2017). Open practice badges. https://www.psychologicalscience.org/publications/badges
2. Blattman, C., (2016, March 27). Preregistration of clinical trials causes medicines to stop working. Chris Blattman. https://chrisblattman.com/2016/03/01/13719/
3. Center for Open Science. (2020). OSF Pre-registration template. https://osf.io/jea94/
4. Cohen, G. L., Garcia, J., Apfel, N., & Master, A., (2006). Reducing the racial achievement gap: A social-psychological intervention. Science, 313 (5791), 1307–1310. https://doi.org/10.1126/science.1128317
5. Cron, L., (2012). Wired for story: The writer's guide to using brain science to hook readers from the very first sentence. (1st ed.). Ten Speed Press.
6. Dee, T. S., (2015). Social identity and achievement gaps: Evidence from an affirmation intervention. Journal of Research on Educational Effectiveness, 8 (2), 149–168. https://doi.org/10.1080/19345747.2014.906009
7. Dominus, S., (2017). Oct. 18). When the revolution came for Amy Cuddy. The New York Times Magazine. https://www.nytimes.com/2017/10/18/magazine/when-the-revolution-came-for-amy-cuddy.html
8. Flake, J. K., (2021/this issue). Strengthening the foundation of educational psychology by integrating construct validation into open science reform. Educational Psychologist, 56 (2), 132–141. https://doi.org/10.1080/00461520.2021.1898962
9. Fleming, J. I., Wilson, S. E., Hart, S. A., Therrien, W. J., & Cook, B. G., (2021/this issue). Open accessibility in education research: Enhancing the credibility, equity, impact, and efficiency of research. Educational Psychologist, 56 (2), 110–121. https://doi.org/10.1080/00461520.2021.1897593
10. Gehlbach, H., (2015). Seven survey sins. The Journal of Early Adolescence, 35 (5–6), 883–897. https://doi.org/10.1177/0272431615578276
11. Gehlbach, H., Brinkworth, M. E., King, A. M., Hsu, L. M., McIntyre, J., & Rogers, T., (2016). Creating birds of similar feathers: Leveraging similarity to improve teacher–student relationships and academic achievement. Journal of Educational Psychology, 108 (3), 342–352. https://doi.org/10.1037/edu0000042
12. Gehlbach, H., & Robinson, C. D., (2018). Mitigating illusory results through preregistration in education. Journal of Research on Educational Effectiveness, 11 (2), 296–315. https://doi.org/10.1080/19345747.2017.1387950
13. Gelman, A., & Loken, E., (2014). The statistical crisis in science. American Scientist, 102 (6), 460–465. https://doi.org/10.1511/2014.111.460
14. Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D., (2016). Comment on “Estimating the reproducibility of psychological science". Science, 351 (6277), 1037–1037. https://doi.org/10.1126/science.aad7243
15. Hanselman, P., Rozek, C. S., Grigg, J., & Borman, G. D., (2017). New evidence on self-affirmation effects and theorized sources of heterogeneity from large-scale replications. Journal of Educational Psychology, 109 (3), 405–424. https://doi.org/10.1037/edu0000141
16. Harackiewicz, J. M., Canning, E. A., Tibbetts, Y., Giffen, C. J., Blair, S. S., Rouse, D. I., & Hyde, J. S., (2014). Closing the social class achievement gap for first-generation students in undergraduate biology. Journal of Educational Psychology, 106 (2), 375–389. https://doi.org/10.1037/a0034679
17. Ioannidis, J. P. A., (2005). Why most published research findings are false. PLOS Medicine, 2 (8), Article e124. https://doi.org/10.1371/journal.pmed.0020124
18. Jacowitz, K. E., & Kahneman, D., (1995). Measures of anchoring in estimation tasks. Personality and Social Psychology Bulletin, 21 (11), 1161–1166. https://doi.org/10.1177/01461672952111004
19. Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., Nosek, B. A., (2014). Investigating variation in replicability. Social Psychology, 45 (3), 142–152. https://doi.org/10.1027/1864-9335/a000178
20. Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., Batra, R., Berkics, M., Bernstein, M. J., Berry, D. R., Bialobrzeska, O., Binan, E. D., Bocian, K., Brandt, M. J., Busching, R., Nosek, B. A., Š. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1 (4), 443–490. https://doi.org/10.1177/2515245918810225
21. Kristal, A. S., Whillans, A. V., Bazerman, M. H., Gino, F., Shu, L. L., Mazar, N., & Ariely, D., (2020). Signing at the beginning versus at the end does not decrease dishonesty. Proceedings of the National Academy of Sciences of the United States of America, 117 (13), 7103–7107. https://doi.org/10.1073/pnas.1911695117
22. Makel, M. C., & Plucker, J. A., (2014). Facts are more important than novelty: Replications in the education sciences. Educational Researcher, 43 (6), 304–316. https://doi.org/10.3102/0013189X14545513
23. Maxwell, S. E., Lau, M. Y., & Howard, G. S., (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? The American Psychologist, 70 (6), 487–498. https://doi.org/10.1037/a0039400
24. Mellor, D., (2021/this issue). Improving norms in research culture to incentivize transparency and rigor. Educational Psychologist, 56 (2), 122–131. https://doi.org/10.1080/00461520.2021.1902329
25. Miyake, A., Kost-Smith, L. E., Finkelstein, N. D., Pollock, S. J., Cohen, G. L., & Ito, T. A., (2010). Reducing the gender achievement gap in college science: A classroom study of values affirmation. Science, 330 (6008), 1234–1237. https://doi.org/10.1126/science.1195996
26. Muenks, K., Wigfield, A., Yang, J. S., & O'Neal, C. R., (2017). How true is grit? Assessing its relations to high school and college students’ personality characteristics, self-regulation, engagement, and achievement. Journal of Educational Psychology, 109 (5), 599–620. https://doi.org/10.1037/edu0000153
27. Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., … Yarkoni, T., (2015). SCIENTIFIC STANDARDS. Promoting an open research culture. Science (New York, N.Y.), 348 (6242), 1422–1425. https://doi.org/10.1126/science.aab2374
28. Nosek, B. A., Banaji, M. R., & Greenwald, A. G., (2002). Harvesting implicit group attitudes and beliefs from a demonstration web site. Group Dynamics: Theory, Research, and Practice, 6 (1), 101–115. https://doi.org/10.1037/1089-2699.6.1.101
29. Nuzzo, R., (2014). Scientific method: statistical errors. Nature, 506 (7487), 150–152. https://doi.org/10.1038/506150a
30. Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349 (6251), 716–711. https://doi.org/10.1126/science.aac4716
31. Patall, E. A., (2021/this issue). Implications of the open science era for educational psychology research syntheses. Educational Psychologist, 56 (2), 142–160. https://doi.org/10.1080/00461520.2021.1897009
32. Phillips, D. C., Floden, R., Gehlbach, H., Lee, C., Warren Little, J., Maynard, R. A., Metz, M. H., Moje, E. B., Sandoval, W. A., Silverman, S., & Soudien, C., (2009). The preparation of aspiring educational researchers in the empirical qualitative and quantitative traditions of social science: Methodological rigor, social and theoretical relevance, and more. Report of Task Force of the Spencer Foundation Educational Research Training Grant Institutions.
33. Plucker, J. A., & Makel, M. C., (2021/this issue). Replication is important for educational psychology: Recent developments and key issues. Educational Psychologist, 56 (2), 90–100. https://doi.org/10.1080/00461520.2021.1895796
34. Reich, J., (2021/this issue). Preregistration and registered reports. Educational Psychologist, 56 (2), 101–109. https://doi.org/10.1080/00461520.2021.1900851
35. Reich, J., Gehlbach, H., & Albers, C., (2020). Like upgrading from a typewriter to a computer: Registered reports in education research. AERA Open. https://doi.org/10.1177/2332858420917640
36. Robinson, C. D., Scott, W., & Gottfried, M. A., (2019). Taking it to the next level: A field experiment to improve instructor-student relationships in college. AERA Open, 5 (1). https://doi.org/10.1177/2332858419839707
37. Rosenthal, R., (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86 (3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
38. Serra-Garcia, M., Hansen, K. T., & Gneezy, U., (2020). Can short psychological interventions affect educational performance? Revisiting the effect of self-affirmation interventions. Psychological Science, 31 (7), 865–872. https://doi.org/10.1177/0956797620923587
39. Sherman, D. K., Hartson, K. A., Binning, K. R., Purdie-Vaughns, V., Garcia, J., Taborsky-Barba, S., Tomassetti, S., Nussbaum, A. D., & Cohen, G. L., (2013). Deflecting the trajectory and changing the narrative: How self-affirmation affects academic performance and motivation under identity threat. Journal of Personality and Social Psychology, 104 (4), 591–618. https://doi.org/10.1037/a0031495
40. Simmons, J. P., Nelson, L. D., & Simonsohn, U., (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22 (11), 1359–1366. https://doi.org/10.1177/0956797611417632
41. Simons, D. J., (2018). Introducing advances in methods and practices in psychological science. Advances in Methods and Practices in Psychological Science, 1 (1), 3–6. https://doi.org/10.1177/2515245918757424
42. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P -curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143 (2), 534–547. https://doi.org/10.1037/a0033242
43. Slavin, R. E., & Lake, C., (2008). Effective programs in elementary mathematics: A best-evidence synthesis. Review of Educational Research, 78 (3), 427–515. https://doi.org/10.3102/0034654308317473
44. Slavin, R. E., Lake, C., Chambers, B., Cheung, A., & Davis, S., (2009). Effective reading programs for the elementary grades: A best-evidence synthesis. Review of Educational Research, 79 (4), 1391–1466. https://doi.org/10.3102/0034654309341374
45. Slavin, R. E., & Smith, D., (2009). The relationship between sample sizes and effect sizes in systematic reviews in education. Educational Evaluation and Policy Analysis, 31 (4), 500–506. https://doi.org/10.3102/0162373709352369
46. van der Zee, T., & Reich, J., (2018). Open education science. AERA Open, 4 (3). https://doi.org/10.1177/2332858418787466
47. Wentzel, K. R., (2021/this issue). Open science reforms: Strengths, challenges, and future directions. Educational Psychologist, 56 (2), 161–173. https://doi.org/10.1080/00461520.2021.1901709