Scientific Utopia: I. Opening Scientific Communication

Psychological Inquiry

Volumn 23, Issue 3, 2012, Pages 217-243

  Nosek, Brian A ^a   Bar Anan, Yoav ^b  

^a UNIVERSITY OF VIRGINIA   (United States)

^b BEN GURION UNIVERSITY OF THE NEGEV   (Israel)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84865736540     PISSN: 1047840X     EISSN: None     Source Type: Journal    
DOI: 10.1080/1047840X.2012.692215     Document Type: Article

References (104)

1. Adler, N. and Harzing, A., W. 2009. When knowledge wins: Transcending the sense and nonsense of academic rankings. The Academy of Management Learning & Education, 8: 72-95.
2. American Chemical Society. 2010. Ensuring access to high-quality science (Public policy statement). Retrieved fromhttp://portal.acs.org/portal/PublicWebSite/policy/publicpolicies/balance/highqualityscience/WPCP_011535
3. American Psychological Association. 2008. Frequently asked questions regarding APA'S ongoing efforts to implement the new public access policy of the National Institutes of Health (NIH). Retrieved fromhttp://www.apa.org/about/gr/issues/access/faq.pdf
4. Antelman, K. 2004. Do open access articles have a greater research impact?. College & Research Libraries News, 65: 372-382.
5. Arms, W. Y. 2002. What are the alternatives to peer review? Quality control in scholarly publishing on the web. The Journal of Electronic Publishing, 8 doi:10.3998/3336451.0008.103
6. Armstrong, J. S. 1997. Peer review for journals: Evidence on quality control, fairness, and innovation. Science and Engineering Ethics, 3: 63-84.
7. Bakker, M. and Wicherts, J. M. 2011. The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43: 666-678. doi:10.3758/s13428-011-0089-5
8. Begg, C. B. 1994. "Publication bias". In Handbook of research synthesis, Edited by: Cooper, H. and Hedges, L. V. 399-409. New York, NY: Russell Sage Foundation.
9. Benos, D. J., Bashari, E., Chaves, J. M., Gaggar, A., Kapoor, N., LaFrance, M. and Zotov, A. 2006. The ups and downs of peer review. Advances in Physiology Education, 31: 145-152. doi:10.1152/advan.00104.2006
10. Berle, D. and Starcevic, V. 2007. Inconsistencies between reported test statistics and p-values in two psychiatry journals. International Journal of Methods in Psychiatric Research, 16: 202-207. doi:10.1002/mpr.225
11. Bielo, D. 2007. Open access to science under attack. Scientific American, January 26 Retrieved fromhttp://www.scientificamerican.com/
12. Bishop, D. 2012. Time for neuroimaging (and PNAS) to clean up its act [Web log post]. BishopBlog, March 5 Retrieved fromhttp://deevybee.blogspot.com/2012/03/time-for-neuroimaging-to-clean-up-its.html.
13. Björk, B-C., Roos, A. and Lauri, M. 2009. Scientific journal publishing: Yearly volume and open access availability. Information Research, 14(1) paper 391. Retrieved fromhttp://InformationR.net/ir/14-1/paper391.html
14. Bor, D. 2012. The dilemma of weak neuroimaging papers [Web log post]. March 8 Retrieved fromhttp://www.danielbor.com/dilemma-weak-neuroimaging/
15. Bornmann, L., Mutz, R. and Daniel, H.-D. 2010. A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants. PloS ONE, 5(12): e14331 doi:10.1371/journal.pone.0014331
16. Brehm, S. 2007. Open access: A thorny debate. APA Monitor on Psychology, 38: 5
17. Budd, J. M., Sievert, M. and Schultz, T. R. 1998. Phenomena of retraction: reasons for retraction and citations to the publications. The Journal of the American Medical Association, 280: 296-297. Retrieved fromhttp://www.ncbi.nlm.nih.gov/pubmed/9676689
18. Cameron, B. D. 2005. Trends in the usage of ISI bibliometric data: Uses, abuses and implications. Portal: Libraries and the Academy, 5: 105-125.
19. Cicchetti, D. V. 1991. The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14: 119-135.
20. Cole, S. 1983. The hierarchy of the sciences?. American Journal of Sociology, 89: 111-139.
21. Davis, P. M., Lewenstein, B. V., Simon, D. H., Booth, J. G. and Connolly, M. J. L. 2008. Open access publishing, article downloads, and citations: Randomised controlled trial. British Medical Journal, 337: a568 doi:10.1136/bmj.a568
22. Egger, M., Davey Smith, G., Schneider, M. and Minder, C. 1997. Bias in meta-analysis detected by a simple, graphical test. British Journal of Medicine, 315(7109): 629-634.
23. eLife. n.d. About. Retrieved fromhttp://www.elifesciences.org/about/
24. The Elsevier boycott and its relationship to WUSM. 2012. Bernard Becker Medical Library. February 28 Retrieved fromhttps://becker.wustl.edu/about/news/elsevier-boycott-and-its-relationship-wusm
25. Eysenbach, G. 2006. Citation advantage of open access articles. PLoS Biology, 4(5): e157 doi:10.1371/journal.pbio.0040157
26. Faculty Advisory Council. 2012. Faculty Advisory Council memorandum on journal pricing, Cambridge, MA: Harvard University. Retrieved fromhttp://isites.harvard.edu/icb/icb.do?keyword=k77982&tabgroupid=icb.tabgroup143448
27. Fanelli, D. 2010. "Positive" results increase down the hierarchy of the sciences. PLoS ONE, 5: e10068 doi:10.1371/ journal.pone.0010068
28. Fanelli, D. 2012. Negative results are disappearing from most disciplines and countries. Scientometrics, 90: 891-904.
29. Fiske, D. W. and Fogg, L. 1990. But the reviewers are making different criticisms of my paper! Diversity and uniqueness in reviewer comments. American Psychologist, 45: 591-598.
30. Garcia-Berthou, E. and Alcaraz, C. 2004. Incongruence between test statistics and p-values in medical papers. BMC Medical Research Methodology, 4: 13 doi:10.1186/1471-2288-4-13
31. Gargouri, Y., Hajjem, C., Larivie're, V., Gingras, Y., Carr, L., Brody, T. and Harnad, S. 2010. Self-selected or mandated, open access increases citation impact for higher quality research. PLoS ONE, 5(10): e13636 doi:10.1371/journal.pone.0013636
32. Giles, J. 2007. PR's "pit bull" takes on open access. Nature, 445: 347 doi:10.1038/445347a
33. Goldbeck-Wood, S. 1999. Evidence on peer review-Scientific quality control or smokescreen?. British Medical Journal, 318: 44 doi:10.1136/bmj.318.7175.44
34. Goodstein, D. 2011. "How science works". In National Research Council reference manual on scientific evidence (3rd ed., 37-54. Washington, DC: The National Academies Press.
35. Google Scholar. 2012. Top publications in English. Retrieved fromhttp://scholar.google.com/citations?view_op=top_venues
36. Greenwald, A. G. 1975. Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82: 1-20.
37. Greenwald, A. G., McGhee, D. E. and Schwartz, J. K. L. 1998. Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74: 1464-1480.
38. Greenwald, A. G., Pratkanis, A. R., Leippe, M. R. and Baumgardner, M. H. 1986. Under what conditions does theory obstruct research progress?. Psychological Review, 93: 216-229.
39. Hardisty, D. J. and Haaga, D. A. F. 2008. Diffusion of treatment research: Does open access matter?. Journal of Clinical Psychology, 64: 821-839.
40. Hargens, L. L. 1988. Scholarly consensus and journal rejection rates. American Sociological Review, 53: 139-151.
41. Harnad, S. 1998. The invisible hand of peer review. Nature, November 5 Retrieved fromhttp://www.nature.com/nature/webmatters/invisible/invisible.html
42. Harnad, S. 2003. Open access to peer-reviewed research through author/institution self-archiving: Maximizing research impact by maximizing online access. Journal of Postgrad Medicine, 49: 337-342.
43. Holden, G., Rosenberg, G., Barker, K. and Onghena, P. 2006. An assessment of the predictive validity of impact factor scores: Implications for academic employment decisions in social work. Research on Social Work Practice, 16: 613-24.
44. Horrobin, D. F. 1990. The philosophical basis of peer review and the suppression of innovation. Journal of the American Medical Association, 263: 1438-1441. doi:10.1001/ jama.1990.03440100162024
45. Houghton, J. 2009. Open access-What are the economic benefits? A comparison of the United Kingdom, Netherlands, and Denmark. Knowledge Exchange,
46. Houghton, J. W., Rasmussen, B., Sheehan, P. J., Oppenheim, C., Morris, A., Creaser, C. and Gourlay, A. 2009. Economic implications of alternative scholarly publishing models: Exploring the costs and benefits, London & Bristol: The Joint Information Systems Committee (JISC) Retrieved fromhttp://www.cfses.com/EI-ASPM/
47. Informa. 2011. Annual report & financial statements 2010, Zug, Switzerland: Author. Retrieved fromhttp://www.informa.com/documents/INF2570%20AR10%20cover%20AW05.pdf
48. Ioannidis, J. P. A. 2005. Why most published research findings are false. PLoS Medicine, 2: e124
49. Ioannidis, J. P. A. 2011. Excess significance bias in the literature on brain volume abnormalities. Archives of General Psychiatry, 68: 773-780. doi:10.1001/archgenpsychiatry.2011.28
50. Jinha, A. E. 2010. Article 50 million: An estimate of the number of scholarly articles in existence. Learned Publishing, 23: 258-263.
51. Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F. and Baker, C. I. 2009. Circular analysis in systems neuroscience: The dangers of double dipping. Nature Neuroscience, 12: 535-540. doi:10.1038/nn.2303.
52. Lerner, J. S. and Tetlock, P. E. 1999. Accounting for the effects of accountability. Psychological Bulletin, 125: 255-275.
53. Marris, E. 2005. American Chemical Society: Chemical reaction. Nature, 437: 807-809. doi:10.1038/437807a
54. Marsh, H. W. and Ball, S. 1989. The peer review process used to evaluate manuscripts submitted to academic journals: Interjudgmental reliability. The Journal of Experimental Education, 57: 151-169.
55. Mathieu, S., Boutron, I., Moher, D., Altman, D. G. and Ravaud, P. 2009. Comparison of registered and published primary outcomes in randomized controlled trials. Journal of the American Medical Association, 302: 977-984. doi:10.1001/jama.2009.1242
56. MIT Libraries. n.d. Elsevier fact sheet. Retrieved fromhttp://libraries.mit.edu/sites/scholarly/mit-open-access/open-access-at-mit/mit-open-access-policy/publishers-and-the-mit-faculty-open-access-policy/elsevier-fact-sheet/
57. Morin, A., Urban, J., Adams, P. D., Foster, I., Sali, A., Baker, D. and Sliz, P. 2012. Shining light into black boxes. Science, 336: 159-160.
58. The Neurocritic. 2012. How much of the neuroimaging literature should we discard? [Web log post]. March 6 Retrieved fromhttp://neurocritic.blogspot.com/2012/03/how-much-of-neuroimaging-literature.html
59. Nielsen, M. 2012. Reinventing discovery, Princeton, NJ: Princeton University Press.
60. Nieuwenhuis, S., Forstmann, B. U. and Wagenmakers, E.-J. 2011. Erroneous analyses of interactions in neuroscience: A problem of significance. Nature Neuroscience, 14: 1105-1107. doi:10.1038/nn.2886
61. Nosek, B. A., Spies, J. R. and Motyl, M. in press. Scientific utopia: II. Restructuring incentives and practices to promote true results, not just publishable results Unpublished manuscript
62. Nosek, B. A., Hawkins, C. B. and Frazier, R. S. 2011. Implicit social cognition: From measures to mechanisms. Trends in Cognitive Sciences, 15: 152-159.
63. OAD. 2012. "Open access directory". Retrieved fromhttp://oad.simmons.edu/oadwiki/Main_Page
64. Of goats and headaches. 2011. The Economist May 26 Retrieved fromhttp://www.economist.com/node/18744177
65. Open access. n.d. Wikipedia Retrieved fromhttp://en.wikipedia.org/wiki/Open_access
66. Open peer-review. n.d. Validation of web-based physical activity measurement systems using doubly labeled water, and other submissions. Journal of Medical Internet Research., Retrieved fromhttp://www.jmir.org/reviewer/openReview/abstracts
67. Peine, M. 2011. 2010 study of subscription prices for scholarly society journals: Society journal pricing trends and industry overview, Lawrence, KS: Allen Press.
68. Peters, D. P. and Ceci, S. J. 1982. Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5: 187-255.
69. Petty, R. E., Fleming, M. A. and Fabrigar, L. R. 1999. The review process at PSPB: Correlates of interreviewer agreement and manuscript acceptance. Personality and Social Psychology Bulletin, 25: 188-203. doi:10.1177/0146167299025002005.
70. Public Library of Science. 2010. Public Library of Science 2010 progress report Retrieved fromhttp://www.plos.org/media/downloads/2011/2010_PLoS_Progress_Update_lo.pdf
71. Public Library of Science. n.d. PLoS ONE journal information. Retrieved fromhttp://www.plosone.org/static/information.action
72. Poldrack, R. 2012. Skeletons in the closet [Web log post]. March 6 Retrieved fromhttp://www.russpoldrack.org/2012/03/skeletons-in-closet.html
73. Redman, B. K., Yarandi, H. N. and Merz, J. F. 2008. Empirical developments in retraction. Journal of Medical Ethics, 34: 807-809. doi:10.1136/jme.2007.023069
74. Reich, D. A., Green, M. C., Brock, T. C. and Tetlock, P. E. 2007. Biases in research evaluation: Inflated assessment, oversight, or error-type weighting?. Journal of Experimental Social Psychology, 43: 633-640.
75. Reichman, O. J., Jones, M. B. and Schildhauer, M. P. 2011. Challenges and opportunities of open data in ecology. Science, 331: 703-705. doi:10.1126/science.1197962
76. Research Information Network. 2008. Activities, costs and funding flows in the scholarly communications system in the UK. Retrieved fromhttp://www.rin.ac.uk/our-work/communicating-and-disseminating-research/activities-costs-and-funding-flows-scholarly-commu
77. Rosenthal, R. 1978. How often are our numbers wrong?. American Psychologist, 33: 1005-1008. doi:10.1037/0003-066X.33.11.1005
78. Rosenthal, R. 1979. The file drawer problem and tolerance for null results. Psychological Bulletin, 86: 638-641.
79. Rossi, J. S. 1987. How often are our statistics wrong? A statistics class exercise. Teaching of Psychology, 14: 98-101. doi:10.1207/s15328023top1402_8
80. Rothstein, H. R., Sutton, A. J. and Borenstein, M. 2005. Publication bias in meta-analysis Prevention, assessment and adjustments, Hoboken, NJ: Wiley.
81. Schooler, J. W. 2011. Unpublished results hide the decline effect. Nature, 470: 437
82. Science. n.d. Wikipedia Retrieved fromhttp://en.wikipedia.org/wiki/Science
83. Seglen, P. O. 1994. Causal relationship between article citedness and journal impact. Journal of the American Society for Information Science, 45: 1-11.
84. Seglen, P. O. 1997. Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314: 498-502.
85. Smith, R. 1999. Opening up BMJ peer review. British Medical Journal, 318: 4 doi:10.1136/bmj.318.7175.4
86. Starbuck, W. H. 2005. How much better are the most-prestigious journals? The statistics of academic publication. Organization Science, 16: 180-200.
87. Sterling, T. D. 1959. Publication decisions and their possible effects on inferences drawn from tests of significance-or vice versa. Journal of the American Statistical Association, 54: 30-34.
88. Sterling, T. D., Rosenbaum, W. L. and Weinkam, J. J. 1995. Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician, 49: 108-112.
89. Stodden, V. 2011. Trust your science? Open your data and code. Amstat News,: 21-22. July
90. Suber, P. 2012. Open access overview. June 18 Retrieved fromhttp://www.earlham.edu/~peters/fos/overview.htm
91. Sunstein, C. 2009. Republic.com 2.0., Princeton, NJ: Princeton University Press.
92. Sutton, A. J., Duval, S. J., Tweedie, R. L., Abrams, K. R. and Jones, D. R. 2000. Empirical assessment of effect of publication bias on meta-analysis. British Journal of Medicine, 320: 1574-1577.
93. Taylor, M. P. 2012. Academic publishing is broken. The Scientist, March 19 Retrieved fromhttp://the-scientist.com/2012/03/19/opinion-academic-publishing-is-broken/
94. Temple, E., Deutsch, G. K., Poldrack, R. A., Miller, S. L., Tallal, P., Merzenich, M. M. and Gabrieli, J. D. E. 2003. Neural deficits in children with dyslexia ameliorated by behavioral remediation: Evidence from functional MRI. Proceedings of the National Academy of Sciences, 100: 2860-2865. doi:10.1073/pnas.0030098100
95. van Rooyen, S., Godlee, F., Evans, S., Black, N. and Smith, R. 1999. Effect of open peer review on quality of reviews and on reviewers' recommendations: A randomised trial. British Medical Journal, 318: 23 doi:10.1136/bmj.318.7175.23
96. Wellcome Trust. n.d. Open access policy. Retrieved fromhttp://www.wellcome.ac.uk/About-us/Policy/Policy-and-position-statements/WTD002766.htm
97. Westberg, L. 2012. Journals maintain scholastic monopoly. The Exponent Online., February 13 Retrieved fromhttp://www.purdueexponent.org/campus/article_19f1031d-0396-5b21-870a-c7f9f6ca91bc.html
98. Whitehurst, G. J. 1984. Interrater agreement for journal manuscript reviews. American Psychologist, 39: 22-28. doi:10.1037//0003-066X.39.1.22
99. Wicherts, J. M. 2011. Psychology must learn a lesson from fraud case. Nature, 480: 7
100. Wicherts, J. M., Borsboom, D., Kats, J. and Molenaar, D. 2006. The poor availability of psychological research data for reanalysis. American Psychologist, 61: 726-728.
101. Wicherts, J. M., Kievit, R. A., Bakker, M. and Borsboom, D. 2012. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Frontiers in Computational Neuroscience, 6: 20 doi:10.3389/ fncom.2012.00020
102. Wilson, T. D., DePaulo, B. M., Mook, D. G. and Klaaren, K. J. 1993. Scientists' evaluations of research: the biasing effects of the importance of the topic. Psychological Science, 4: 322-325.
103. Yarkoni, T., Poldrack, R. A., Van Essen, D. C. and Wager, T. D. 2010. Cognitive neuroscience 2.0: Building a cumulative science of human brain function. Trends in Cognitive Sciences, 14: 489-496.
104. Zuckerman, H. A. and Merton, R. K. 1971. Patterns of evaluation in science: Institutionalization, structure and functions of the referee system. Minerva, 9: 66-100.