Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid P-hacking

Frontiers in Psychology

Volumn 7, Issue NOV, 2016, Pages

  Wicherts, Jelte M ^a   Veldkamp, Coosje L S ^a   Augusteijn, Hilde E M ^a   Bakker, Marjan ^a   van Aert, Robbie C M ^a   van Assen, Marcel A L M ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

Bias; Experimental design; P hacking; Questionable research practices; Research methods education; Significance chasing; Significance testing

Indexed keywords

EID: 85006415925     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2016.01832     Document Type: Review

References (57)

1. Asendorpf, J. B., Conner, M., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., et al. (2013). Recommendations for increasing replicability in psychology. Eur. J. Personal. 27, 108-119. doi: 10.1002/per.1919.
2. Bakker, M., Hartgerink, C. H., Wicherts, J. M., and van der Maas, H. L. (2016). Researchers' Intuitions about power in psychological research. Psychol. Sci. 27, 1069-1077. doi: 10.1177/0956797616647519.
3. Bakker, M., van Dijk, A., and Wicherts, J. M. (2012). The rules of the game called psychological science. Perspect. Psychol. Sci. 7, 543-554. doi: 10.1177/1745691612459060.
4. Bakker, M., and Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behav. Res. Methods 43, 666-678. doi: 10.3758/s13428-011-0089-5.
5. Bakker, M., and Wicherts, J. M. (2014). Outlier removal, sum scores, and the inflation of the Type I error rate in independent samples t tests. The power of alternatives and recommendations. Psychol. Methods 19, 409-427. doi: 10.1037/met0000014.
6. Bargh, J. A., and Chartrand, T. L. (2000). "Studying the mind in the middle; A practical guide to priming and automaticity research," in Handbook of Research Methods in Social and Personality Psychology, eds H. Reis and C. Judd (Cambridge: Cambridge University Press), 253-285.
7. Barnett, V., and Lewis, T. (1994). Outliers in Statistical Data. Chichester: John Wiley & Sons.
8. Chambers, C. D. (2013). Registered reports: a new publishing initiative at Cortex. Cortex 49, 609-610. doi: 10.1016/j.cortex.2012.12.016.
9. Chan, A.-W., Hrobjartsson, A., Haahr, M. T., Gotzsche, P. C., and Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials-Comparison of Protocols to published articles. JAMA 291, 2457-2465. doi: 10.1001/jama.291.20.2457.
10. Cohen, J. (1990). Things I have learned (thus far). Am. Psychol. 45, 1304-1312. doi: 10.1037/0003-066X.45.12.1304.
11. Cooper, H., DeNeve, K., and Charlton, K. (1997). Finding the missing science: the fate of studies submitted for review by a human subjects committee. Psychol. Methods 2, 447-452. doi: 10.1037/1082-989X.2.4.447.
12. Cumming, G. (2012). Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis. New York, NY: Routledge.
13. Cumming, G. (2014). The new statistics: why and how. Psychol. Sci. 25, 7-29. doi: 10.1177/0956797613504966.
14. de Groot, A. D. (1956/2014). The meaning of "significance" for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. 1969. Acta Psychol. 148, 188-194. doi: 10.1016/j.actpsy.2014.02.001.
15. DeCoster, J., Sparks, E. A., Sparks, J. C., Sparks, G. G., and Sparks, C. W. (2015). Opportunistic biases: their origins, effects, and an integrated solution. Am. Psychol. 70, 499-514. doi: 10.1037/a0039191.
16. Eich, E. (2014). Business not as usual. Psychol. Sci. 25, 3-6. doi: 10.1177/0956797613512465.
17. Francis, G. (2013). Replication, statistical consistency, and publication bias. J. Math. Psychol. 57, 153-169. doi: 10.1016/j.jmp.2013.02.003.
18. Franco, A., Malhotra, N., and Simonovits, G. (2016). Underreporting in psychology experiments: evidence from a study registry. Soc. Psychol. Personal. Sci. 7, 8-12. doi: 10.1177/1948550615598377.
19. Gelman, A., and Loken, E. (2014). The statistical crisis in science. Am. Sci. 102, 460-465. doi: 10.1511/2014.111.460.
20. Hubbard, R. (2015). Corrupt Research: the Case for Reconceptualizing Empirical Management and Social Science. Thousand Oaks, CA: SAGE Publications.
21. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Med. 2:e124. doi: 10.1371/journal.pmed.0020124.
22. Ioannidis, J. P. A. (2008). Why most discovered true associations are inflated. Epidemiology 19, 640-648. doi: 10.1097/EDE.0b013e31818131e7.
23. John, L. K., Loewenstein, G., and Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychol. Sci. 23, 524-532. doi: 10.1177/0956797611430953.
24. Kerr, N. L. (1998). HARKing: hypothesizing after the results are known. Pers. Soc. Psychol. Rev. 2, 196-217. doi: 10.1207/s15327957pspr0203_4.
25. Kidwell, M. C., Lazarevic, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L. S., et al. (2016). Badges to acknowledge open practices: a simple, low-cost, effective method for increasing transparency. PLoS Biol. 14:e1002456. doi: 10.1371/journal.pbio.1002456.
26. Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., et al. (2010). The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 340:c365. doi: 10.1136/bmj.c365.
27. Kriegeskorte, N., Lindquist, M. A., Nichols, T. E., Poldrack, R. A., and Vul, E. (2010). Everything you never wanted to know about circular analysis, but were afraid to ask. J. Cereb. Blood Flow Metab. 30, 1551-1557. doi: 10.1038/jcbfm.2010.86.
28. Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., and Baker, C. I. (2009). Circular analysis in systems neuroscience: the dangers of double dipping. Nat. Neurosci. 12, 535-540. doi: 10.1038/nn.2303.
29. Kruschke, J. K. (2015). Doing Bayesian Data Analysis: A Tutorial Introduction with R, JAGS and Stan. London: Academic Press.
30. LeBel, E. P., Borsboom, D., Giner-Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., et al. (2013). PsychDisclosure. org grassroots support for reforming reporting standards in psychology. Perspect. Psychol. Sci. 8, 424-432. doi: 10.1177/1745691613491437.
31. Maxwell, S. E., Lau, M. Y., and Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does "failure to replicate" really mean? Am. Psychol. 70, 487-498. doi: 10.1037/a0039400.
32. Nieuwenhuis, S., Forstmann, B. U., and Wagenmakers, E. J. (2011). Erroneous analyses of interactions in neuroscience: a problem of significance. Nat. Neurosci. 14, 1105-1107. doi: 10.1038/nn.2886.
33. Nosek, B. A., Alter, G., Banks, G., Borsboom, D., Bowman, S., Breckler, S., et al. (2015). Promoting an open research culture: author guidelines for journals could help to promote transparency, openness, and reproducibility. Science 348, 1422-1425. doi: 10.1126/science.aab2374.
34. Nosek, B. A., Spies, J., and Motyl, M. (2012). Scientific Utopia: II-Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615-631. doi: 10.1177/1745691612459058.
35. Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., and Wicherts, J. M. (2015). The prevalence of statistical reporting errors in psychology (1985-2013). Behav. Res. Methods doi: 10.3758/s13428-015-0664-2 [Epub ahead of print].
36. Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science 349:aac4716. doi: 10.1126/science.aac4716.
37. Oppenheimer, D. M., Meyvis, T., and Davidenko, N. (2009). Instructional manipulation checks: detecting satisficing to increase statistical power. J. Exp. Soc. Psychol. 45, 867-872. doi: 10.1016/j.jesp.2009.03.009.
38. Poldrack, R., Baker, C. I., Durnez, J., Gorgolewski, K., Matthews, P. M., Munafo, M., et al. (2016). Scanning the Horizon: towards transparent and reproducible neuroimaging research. bioRxiv doi: 10.1101/05918.
39. Rosenthal, R. (1966). Experimenter Effects in Behavioral Research. New York, NY: Appleton-Century-Crofts.
40. Sala-I-Martin, X. X. (1997). I just ran two million regressions. Am. Econ. Rev. 87, 178-183.
41. Schafer, J. L., and Graham, J. W. (2002). Missing data: our view of the state of the art. Psychol. Methods 7, 147-177. doi: 10.1037/1082-989x.7.2.147.
42. Schaller, M. (2016). The empirical benefits of conceptual rigor: systematic articulation of conceptual hypotheses can reduce the risk of non-replicable results (and facilitate novel discoveries too). J. Exp. Soc. Psychol. 66, 107-115. doi: 10.1016/j.jesp.2015.09.0096.
43. Sedlmeier, P., and Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychol. Bull. 105, 309-316. doi: 10.1037/0033-2909.105.2.309.
44. Simmons, J. P., Nelson, L. D., and Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359-1366. doi: 10.1177/0956797611417632.
45. Simons, D. J., Holcombe, A. O., and Spellman, B. A. (2014). An introduction to registered replication reports at perspectives on psychological science. Perspect. Psychol. Sci. 9, 552-555. doi: 10.1177/1745691614543974.
46. Simonsohn, U., Nelson, L. D., and Simmons, J. P. (2014). p-curve and effect size. Correcting for publication bias using only significant results. Perspect. Psychol. Sci. 9, 666-681. doi: 10.1177/1745691614553988.
47. Steegen, S., Tuerlinckx, F., Gelman, A., and Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspect. Psychol. Sci. 11, 702-712. doi: 10.1177/1745691616658637.
48. Ueno, T., Fastrich, G. M., and Murayama, K. (2016). Meta-analysis to integrate effect sizes within an article: possible misuse and Type I error inflation. J. Exp. Psychol. Gen. 145, 643-654. doi: 10.1037/xge0000159.
49. van Aert, R. C. M., Wicherts, J. M., and van Assen, M. A. L. M. (2016). Conducting meta-analyses based on p-values: reservations and recommendations for applying p-uniform and p-curve. Perspect. Psychol. Sci. 11, 713-729. doi: 10.1177/1745691616650874.
50. Wagenmakers, E. J. (2007). A practical solution to the pervasive problems of p values. Psychon. Bull. Rev. 14, 779-804. doi: 10.3758/BF03194105.
51. Wagenmakers, E. J., Wetzels, R., Borsboom, D., Maas, H. L. J. V. D., and Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspect. Psychol. Sci. 7, 632-638. doi: 10.1177/1745691612463078.
52. Wagenmakers, E. J., Wetzels, R., Borsboom, D., and van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). J. Pers. Soc. Psychol. 100, 426-432. doi: 10.1037/a0022790.
53. Weston, J., Dwan, K., Altman, D., Clarke, M., Gamble, C., Schroter, S., et al. (2016). Feasibility study to examine discrepancy rates in prespecified and reported outcomes in articles submitted to The BMJ. BMJ Open 6:e010075. doi: 10.1136/bmjopen-2015-010075.
54. Wicherts, J. M. (2017). "Data re-analysis and open data," in Toward a More Perfect Psychology: Improving Trust, Accuracy, and Transparency in Research, eds J. Plucker and M. Makel (Wahington, DC: American Psychological Association).
55. Wicherts, J. M., Bakker, M., and Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE 6:e26828. doi: 10.1371/journal.pone.0026828.
56. Wigboldus, D. H., and Dotsch, R. (2016). Encourage playing with data and discourage questionable reporting practices. Psychometrika 81, 27-32. doi: 10.1007/s11336-015-9445-1.
57. Wilcox, R. (2012). Modern Statistics for the Social and Behavioral Sciences: A Practical Introduction. Boca Raton, FL: CRC Press.