Measuring the effectiveness of scientific gatekeeping

Proceedings of the National Academy of Sciences of the United States of America

Volumn 112, Issue 2, 2015, Pages 360-365

  Siler, Kyle ^a   Lee, Kirby ^b   Bero, Lisa ^c  

^a UNIVERSITY OF TORONTO   (Canada)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c UNIVERSITY OF SYDNEY   (Australia)

Author keywords

Creativity; Decision making; Innovation; Peer review; Publishing

Indexed keywords

ARTICLE; AWARDS AND PRIZES; BEHAVIORAL ECONOMICS; EDITOR; FUNDING; HALO EFFECT; HUMAN; JOURNAL IMPACT FACTOR; PEER REVIEW; PRIORITY JOURNAL; PUBLICATION; PUBLISHING; SCIENTIFIC LITERATURE; SCIENTIST; SOCIAL STATUS; BIBLIOGRAPHIC DATABASE; DECISION MAKING; PROCEDURES; STANDARDS;

DATABASES, BIBLIOGRAPHIC; DECISION MAKING; EDITORIAL POLICIES; PEER REVIEW, RESEARCH; PUBLISHING;

EID: 84920982313     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.1418218112     Document Type: Article

References (57)

1. Merton RK (1968) The Matthew Effect in science. Science 159(3810):56-63.
2. Lamont M (2009) How Professors Think: Inside the Curious World of Academic Judgment (Harvard Univ Press, Cambridge, MA).
3. Kassirer JP, Campion EW (1994) Peer review. Crude and understudied, but indispensable. JAMA 272(2):96-97.
4. Mahoney MJ (1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognit Ther Res 1(2):161-175.
5. Horrobin DF (1990) The philosophical basis of peer review and the suppression of innovation. JAMA 263(10):1438-1441.
6. Yalow RS (1977) Nobel Lecture. Radioimmunoassay: A Probe For Fine Structure Of Biologic Systems. December 8, 1977 at Stockholm Concert Hall, Stockholm, Sweden.
7. Gans JS, Shepherd GB (1994) How are the mighty fallen: Rejected classic articles by leading economists. J Econ Perspect 8(1):165-179.
8. Nature (2003) Coping with peer rejection. Nature 425(6959):645.
9. Lotka AJ (1926) The frequency distribution of scientific productivity. J Wash Acad Sci 16:317-323.
10. Barabási AL, Albert R (1999) Emergence of scaling in random networks. Science 286(5439):509-512.
11. Lynn FB (2014) Diffusing through disciplines: Insiders, outsiders and socially influenced citation behavior. Soc Forces 93(1):355-382.
12. Hargens LL (2000) Using the literature: Reference networks, reference contexts, and the social structure of scholarship. Am Sociol Rev 65(6):846-865.
13. Bornmann L, Daniel H-D (2008) What do citation counts measure: A review of studies on citing behavior. J Doc 64(1):45-80.
14. Latour B, Woolgar S (1979) Laboratory Life: The Construction of Scientific Facts (Sage, Los Angeles).
15. Ellison G (2002) Evolving standards for academic publishing: A q-r theory. J Polit Econ 110(5):994-1034.
16. Goodman SN, Berlin J, Fletcher SW (1994) Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Annals of Internal Medicine 121(1): 11-21.
17. Lindsey D (1988) Assessing precision in the manuscript review process: A little better than a dice roll. Scientometrics 14(1-2):75-82.
18. Cicchetti DV (1991) The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behav Brain Sci 14(1):119-135.
19. Peters DP, Ceci SJ (1982) Peer review practices of psychological journals: The fate of published articles, submitted again. Behav Brain Sci 5(2):187-195.
20. Leahey E (2004) The role of status in evaluating research: The case of data editing. Soc Sci Res 33(3):521-537.
21. Long JS, Fox MF (1995) Scientific careers: Universalism and particularism. Annu Rev Sociol 24:45-71.
22. Lee CJ, Sugimoto CR, Zhang G, Cronin B (2013) Bias in peer review. JASIST 64(1):2-17.
23. Armstrong JS, Hubbard R (1991) Does the need for agreement among reviewers inhibit the publication of controversial findings? Behav Brain Sci 14(1):136-137.
24. Dale SB, Krueger AB (2002) Estimating the payoff to attending a more selective college: An application of selection on observables and unobservables. Q J Econ 117(4):1491-1527.
25. Alberts B, Kirschner MW, Tilghman S, Varmus H (2014) Rescuing US biomedical research from its systemic flaws. Proc Natl Acad Sci USA 111(16):5773-5777.
26. Gieryn TF (1983) Boundary-work and the demarcation of science. Am Sociol Rev 48(6): 781-795.
27. Abbott AD (1981) Status and status strain in the professions. Am J Sociol 86(4): 819-835.
28. Smith PK, Trope Y (2006) You focus on the forest when you're in charge of the trees: Power priming and abstract information processing. J Pers Soc Psychol 90(4):578-596.
29. Podolny JM (2005) Status Signals: A Sociological Study of Market Competition (Princeton Univ Press, Princeton, NJ).
30. Cole S (1983) The hierarchy of the sciences? Am J Sociol 89(1):111-139.
31. Kuhn TS (1962) The Structure of Scientific Revolutions (Univ of Chicago Press, Chicago).
32. Akerlof GA (2003) Writing "The Market for 'Lemons'": A Personal and Interpretive Essay. Available at www.nobelprize. org/nobel-prizes/economic-sciences/laureates/2001/akerlof-article. html. Accessed June 2, 2014.
33. Starbuck WH (2003) Turning lemons into lemonade: Where is the value in peer reviews? J Manage Inq 12(4):344-351.
34. Available at scatter. files. wordpress. com/2014/10/granovetter-rejection. pdf. Accessed October 23, 2014.
35. McGinty S (June 16, 2003) The J. K. Rowling Story. The Scotsman. Available at www.scotsman. com/lifestyle/books/the-jk-rowling-story-1-652114. Accessed December 9, 2014.
36. Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P, Bero LA (2006) Predictors of publication: Characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med J Aust 184(12):621-626.
37. Dickersin K (1990) The existence of publication bias and risk factors for its occurrence. JAMA 263(10):1385-1389.
38. Ioannidis JPA (2005) Why most published research findings are false. PLoS Medicine 2(8):0696-0701.
39. Calcagno VE, et al. (2012) Flows of research manuscripts among scientific journals reveal hidden submission patterns. Science 388(6110):1065-1069.
40. Kahneman D (2011) Thinking Fast and Slow (Farrar, Straus and Giroux, New York).
41. Mueller JS, Melwani S, Goncalo JA (2012) The bias against creativity: Why people desire but reject creative ideas. Psychol Sci 23(1):13-17.
42. GranovetterM, Soong R (1986) Threshold models of interpersonal effects in consumer demand. J Econ Behav Organ 7:83-99.
43. Arthur WB (1989) Competing technologies, increasing returns and lock-in by historical events. The Economic Journal 99:116-131.
44. Denrell J, Liu C (2012) Top performers are not the most impressive when extreme performance indicates unreliability. Proc Natl Acad Sci USA 109(24):9331-9336.
45. Salganik MJ, Dodds PS, Watts DJ (2006) Experimental study of inequality and unpredictability in an artificial cultural market. Science 311(5762):854-856.
46. Seglen PO (1997) Why the impact factor of journals should not be used for evaluating research. BMJ 314(7079):498-502.
47. Silver N (2012) The Signal and the Noise (Penguin, New York).
48. Massey C, Thaler RH (2013) The loser's curse: Decision making and market efficiency in the National Football League draft. Manage Sci 59(7):1479-1495.
49. Mauboussin MJ (2012) The Success Equation (Harvard Business Review Press, Boston).
50. Dempster AM, ed (2014) Risk and Uncertainty in the Art World (Bloomsbury, London).
51. Gompers P, Lerner J (2006) The Venture Capital Cycle (MIT Press, Cambridge, MA).
52. Hirsch PM (1972) Processing fads and fashions: An organization-set analysis of cultural industry systems. Am J Sociol 77(4):639-659.
53. McAfee RP (2014) Edifying editing. Secrets of Economics Editors, eds Szenberg M, Ramrattan L (MIT Press, Cambridge, MA), pp 33-44.
54. Denrell J, March JG (2001) Adaptation as information restriction: The hot stove effect. Organ Sci 12(5):523-538.
55. Levinthal DA, March JG (1993) The myopia of learning. Strateg Manage J 14(S2): 95-112.
56. Cole S, Cole JR, Simon GA (1981) Chance and consensus in peer review. Science 214(4523):881-886.
57. Stinchcombe AL, Ofshe R (1969) On journal editing as a probabilistic process. Am Sociol 4(2):116-117.