Journal data sharing policies and statistical reporting inconsistencies in psychology

Collabra: Psychology

Volumn 3, Issue 1, 2017, Pages

  Nuijten, Michèle B ^a   Borghuis, Jeroen ^a   Veldkamp, Coosje L S ^a   Dominguez Alvarez, Linda ^b   Van Assen, Marcel A L M ^a,c   Wicherts, Jelte M ^a  

^a TILBURG UNIVERSITY   (Netherlands)

^b ECORYS Netherlands   (Netherlands)

^c UTRECHT UNIVERSITY   (Netherlands)

Author keywords

Data sharing; Journal policy; Meta research; Statistical errors

Indexed keywords

EID: 85049801491     PISSN: None     EISSN: 24747394     Source Type: Journal    
DOI: 10.1525/collabra.102     Document Type: Article

References (53)

1. Agnoli, F., Wicherts, J. M., Veldkamp, C. L. S., Albiero, P., & Cubelli, R. (2017). Questionable research practices among Italian research psychologists. PLoS One, 12(3). DOI: https://doi.org/10.1371/journal.pone.0172792
2. Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H., & Ioannidis, J. P. A. (2011). Public availability of published research data in high-impact journals. PLoS One, 6(9), e24357. DOI: https://doi.org/10.1371/journal.pone.0024357
3. American Psychological Association. (2010). Publication Manual of the American Psychological Association. Sixth Edition. Washington, DC: American Psychological Association.
4. Anagnostou, P., Capocasa, M., Milia, N., Sanna, E., Battaggia, C., Luzi, D., & Bisol, G. D. (2015). When Data Sharing Gets Close to 100%: What Human Paleogenetics Can Teach the Open Science Movement. PLoS One, 10(3). DOI: https://doi.org/10.1371/journal.pone.0121409
5. Bakker, M., & Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43(3), 666–678. DOI: https://doi.org/10.3758/s13428-011-0089-5
6. Bakker, M., & Wicherts, J. M. (2014). Outlier removal and the relation with reporting errors and quality of research. PLoS One, 9(7), e103360. DOI: https://doi.org/10.1371/journal.pone.0103360
7. Barber, L. K. (2017). Meticulous manuscripts, messy results: Working together for robust science reporting. Stress and Health, 33(2), 89–91. DOI: https://doi.org/10.1002/smi.2756
8. Baron, J. (2011). Acknowledgements and report for the year 2010. Judgment and Decision Making, 6(2), 1–3.
9. Berle, D., & Starcevic, V. (2007). Inconsistencies between reported test statistics and p-values in two psychiatry journals. International Journal of Methods in Psychiatric Research, 16(4), 202–207. DOI: https://doi.org/10.1002/mpr.225
10. Bloom, T., Ganley, E., & Winker, M. (2014). Data access for the open access literature: PLOS’s data policy. PLoS biology, 12(2), e1001797. DOI: https://doi.org/10.1371/journal.pbio.1001797
11. Caperos, J. M., & Pardo, A. (2013). Consistency errors in p-values reported in Spanish psychology journals. Psicothema, 25(3), 408–414. DOI: https://doi.org/10.7334/psicothema2012.207
12. Ceci, S. J. (1988). Scientists Attitudes toward Data Sharing. Science Technology & Human Values, 13(1–2), 45–52.
13. Chamberlain, S., Boettiger, C., & Ram, K. (2014). rplos: Interface to PLoS Journals search API. R package version 0.4.0. http://CRAN.R-project.org/package=rplos.
14. Chambers, C. (2017). The seven deadly sins of psychology: A manifesto for reforming the culture of scientific practice. Princeton University Press. DOI: https://doi.org/10.1515/9781400884940
15. Cumming, G., Fidler, F., Leonard, M., Kalinowski, P., Christiansen, A., Kleinig, A., Wilson, S., et al. (2007). Statistical reform in psychology: Is anything changing? Psychological science, 18(3), 230–232. DOI: https://doi.org/10.1111/j.1467-9280.2007.01881.x
16. Eich, E. (2014). Business not as usual. Psychological science, 25(1), 3–6. DOI: https://doi.org/10.1177/0956797613512465
17. Epskamp, S., & Nuijten, M. B. (2014). statcheck: Extract statistics from articles and recompute p values. R package version 1.0.0. http://CRAN.R-project.org/package=statcheck.
18. Epskamp, S., & Nuijten, M. B. (2015). statcheck: Extract statistics from articles and recompute p values. R package version 1.0.1. http://CRAN.R-project.org/package=statcheck.
19. Epskamp, S., & Nuijten, M. B. (2016). statcheck: Extract statistics from articles and recompute p values. R package version 1.2.2. http://CRAN.R-project.org/package=statcheck.
20. Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015). Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science. Journal of Personality and Social Psychology, 108(2), 275–297. DOI: https://doi.org/10.1037/pspi0000007
21. Garcia-Berthou, E., & Alcaraz, C. (2004). Incongruence between test statistics and P values in medical papers. Bmc Medical Research Methodology, 4(1), 13. DOI: https://doi.org/10.1186/1471-2288-4-13
22. Giofrè, D., Cumming, G., Fresc, L., Boedker, I., & Tressoldi, P. (2017). The influence of journal submission guidelines on authors’ reporting of statistics and use of open research practices. PLoS One, 12(4), e0175583. DOI: https://doi.org/10.1371/journal.pone.0175583
23. Hedrick, T. E. (1985). Justifications for and obstacles to data sharing. Sharing research data, 123–147.
24. Hubbard, R., & Ryan, P. A. (2000). The historical growth of statistical significance testing in psychology–and its future prospects. Educational and Psychological Measurement, 60, 661–681. DOI: https://doi.org/10.1177/0013164400605001
25. Ioannidis, J. P. A., Fanelli, D., Dunne, D. D., & Goodman, S. N. (2015). Meta-research: Evaluation and Improvement of Research Methods and Practices. PLoS biology, 13(10), e1002264–e1002264. DOI: https://doi.org/10.1371/journal.pbio.1002264
26. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth–telling. Psychological science, 23, 524–532. DOI: https://doi.org/10.1177/0956797611430953
27. Kidwell, M. C., Lazarevic, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Nosek, B. A., et al. (2016). Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency. PLoS biology, 1–15. DOI: https://doi.org/10.1371/journal.pbio.1002456
28. Krawczyk, M., & Reuben, E. (2012). (Un)Available upon Request: Field Experiment on Researchers’ Willingness to Share Supplementary Materials. Accountability in Research: Policies and Quality Assurance, 19, 175–186. DOI: https://doi.org/10.1080/08989621.2012.678688
29. Lindsay, D. S. (2017). Sharing Data and Materials in Psychological Science. Psychological science, 28(6), 699–702. DOI: https://doi.org/10.1177/0956797617704015
30. Longo, D. L., & Drazen, J. M. (2016). Data sharing. The New England Journal of Medicine, 374, 276–277. DOI: https://doi.org/10.1056/NEJMe1516564
31. Morris, P. E., & Fritz, C. O. (2017). Meeting the challenge of the Psychonomic Society’s 2012 Guidelines on Statistical Issues: Some success and some room for improvement. Psychonomic Bulletin & Review, 1–7. DOI: https://doi.org/10.3758/s13423-017-1267-y
32. Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., du Sert, N. P., Ioannidis, J. P., et al. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. DOI: https://doi.org/10.1038/s41562-016-0021
33. Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Yarkoni, T., et al. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425. DOI: https://doi.org/10.1126/science.aab2374
34. Nuijten, M. B., Hartgerink, C. H. J., Van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 48(4), 1205–1226. DOI: https://doi.org/10.3758/ s13428-015-0664-2
35. Nuijten, M. B., Van Assen, M. A. L. M., Hartgerink, C. H. J., Epskamp, S., & Wicherts, J. M. (2017). The validity of the tool “statcheck” in discovering statistical reporting inconsistencies. Preprint retrieved from: https://psyarxiv.com/tcxaj/.
36. Piwowar, H. A., Day, R. S., & Fridsma, D. B. (2007). Sharing detailed research data is associated with increased citation rate. PLoS One, 2(3), e308. DOI: https://doi.org/10.1371/journal.pone.0000308
37. Sakaluk, J., Williams, A., & Biernat, M. (2014). Analytic Review as a Solution to the Misreporting of Statistical Results in Psychological Science. Perspectives on Psychological Science, 9(6), 652–660. DOI: https://doi.org/10.1177/1745691614549257
38. Schönbrodt, F. D. (2015). p-checker: One-for-all p-value analyzer. Retrieved from: http://shinyapps.org/apps/p-checker/.
39. Shrout, P. E., & Rodgers, J. L. (2017). Psychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis. Annual Review of Psychology, 69(1).
40. Simonsohn, U. (2013). Just post it: The lesson from two cases of fabricated data detected by statistics alone. Psychological science, 24(10), 1875–1888. DOI: https://doi.org/10.1177/0956797613480366
41. Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance – Or vice versa. Journal of the American Statistical Association, 54, 30–34. DOI: https://doi.org/10.2307/2282137
42. Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited – The effect of the outcome of statistical tests on the decision to publish and vice-versa. American Statistician, 49(1), 108–112. DOI: https://doi.org/10.2307/2684823
43. Sterling, T. D., & Weinkam, J. J. (1990). Sharing Scientific-Data. Communications of the Acm, 33(8), 112–119. DOI: https://doi.org/10.1145/79173.791822
44. Vanpaemel, W., Vermorgen, M., Deriemaecker, L., & Storms, G. (2015). Are we wasting a good crisis? The availability of psychological research data after the storm. Collabra, 1(1), 1–5. DOI: https://doi.org/10.1525/collabra.13
45. Veldkamp, C. L. S., Hartgerink, C. H. J., Van Assen, M. A.L.M.,&Wicherts,J.M.(2017).Sharedresponsibility for statistical analyses and statistical Reporting errors in psychology articles published in PLOS ONE (2003–2016). Retrieved from: https://psyarxiv.com/g8cjq.
46. Veldkamp, C. L. S., Nuijten, M. B., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2014). Statistical reporting errors and collaboration on statistical analyses in psychological science. PLoS One, 9(12), e114876. DOI: https://doi.org/10.1371/journal. pone.0114876
47. Vines, T. H., Albert, A. Y., Andrew, R. L., Débarre, F., Bock, D. G., Franklin, M. T., Rennison, D. J., et al. (2014). The availability of research data declines rapidly with article age. Current biology, 24(1), 94–97. DOI: https://doi.org/10.1016/j.cub.2013.11.014
48. Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. DOI: https://doi.org/10.1177/1745691612463078
49. Wicherts, J. M. (2011). Psychology must learn a lesson from fraud case. Nature, 480, 7. DOI: https://doi.org/10.1038/480007a
50. Wicherts, J. M. (2013). Science revolves around the data. Journal of Open Psychology Data, 1(1), e1. DOI: https://doi.org/10.5334/jopd.e1
51. Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS One, 6(11), e26828. DOI: https://doi.org/10.1371/journal.pone.0026828
52. Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726–728. DOI: https://doi.org/10.1037/0003-066X.61.7.726
53. Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Bourne, P. E., et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific data, 3, 160018. DOI: https://doi.org/10.1038/sdata.2016.18