The influence of peer reviewer expertise on the evaluation of research funding applications

PLoS ONE

Volumn 11, Issue 10, 2016, Pages

  Gallo, Stephen A ^a   Sullivan, Joanne H ^a   Glisson, Scott R ^a  

^a American Institute of Biological Sciences   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLINICAL TRIAL; CONTROLLED CLINICAL TRIAL; CONTROLLED STUDY; EVALUATION RESEARCH; FUNDING; HUMAN; MEDICAL RESEARCH; REGRESSION ANALYSIS; SINGLE BLIND PROCEDURE; SOCIAL NETWORK; FEMALE; FINANCIAL MANAGEMENT; MALE; PEER REVIEW; RETROSPECTIVE STUDY;

FEMALE; HUMANS; MALE; PEER REVIEW, RESEARCH; RESEARCH SUPPORT AS TOPIC; RETROSPECTIVE STUDIES;

EID: 84992386211     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0165147     Document Type: Article

References (37)

1. Olbrecht M, Bornmann L. Panel peer review of grant applications: what do we know from research in social psychology on judgment and decision-making in groups? Research Evaluation 2010 19(4): 293-304.
2. Langfeldt L. The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science 2001 31(6): 820-841.
3. Marsh HW, Jayasinghe UW, Bond NW. Improving the peer-review process for grant applications: Reliability, validity, bias, and generalizability. Amer. Psychologist 2008 63(3):160-168.
4. Ginther DK, Schaffer WT, Schnell J, Masimore B, Liu F, Haak LL et al. Race, ethnicity, and NIH research awards. Science 2011 333(6045): 1015-1019. doi: 10.1126/science.1196783 PMID: 21852498
5. Bornmann L, Daniel HD. Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees' decisions. Scientometrics 2005 63(2): 297-320.
6. Danthi N, Wu CO, Shi P, Lauer M. Percentile Ranking and Citation Impact of a Large Cohort of National Heart, Lung, and Blood Institute-Funded Cardiovascular R01 Grants. Circulation research 2014 114(4): 600-606. doi: 10.1161/CIRCRESAHA.114.302656 PMID: 24406983
7. Gallo SA, Carpenter AS, Irwin D, McPartland CD, Travis J, Reynders S, et al. The validation of peer review through research impact measures and the implications for funding strategies PLOS ONE 2014 9(9): e106474. doi: 10.1371/journal.pone.0106474 PMID: 25184367
8. Cole S, Cole JR, Simon GA. Chance and consensus in peer review. Science 1981 214(4523): 881-886. PMID: 7302566
9. Cicchetti DV. The reliability of peer review for manuscript and grant submissions: A crossdisciplinary investigation. Behavioral and Brain Sciences 1991 14(1): 119-186.
10. Fogelholm M, Leppinen S, Auvinen A, Raitanen J, Nuutinen A, Väänänen K. Panel discussion does not improve reliability of peer review for medical research grant proposals. Journal of clinical epidemiology 2012 65(1): 47-52. doi: 10.1016/j.jclinepi.2011.05.001 PMID: 21831594
11. Lee CJ, Sugimoto CR, Zhang G, Cronin B. Bias in peer review. Journal of the American Society for Information Science and Technology 2013 64(1): 2-17.
12. Salas E, Rosen MA, Diaz-Granados D. Expertise-based intuition and decision making in organizations. Journal of Management 2010 36: 941-973.
13. Phillips JK, Klein G, Sieck WR. Expertise in judgment and decision making: A case for training intuitive decision skills. In: Koehler DJ and Harvey N, Blackwell Handbook of Judgment and Decision Making. Oxford: Blackwell 2004 297-315.
14. Dijkstra KA, Pligt J, Kleef GA. Deliberation versus intuition: Decomposing the role of expertise in judgment and decision making. Journal of Behavioral Decision Making 2013 26(3): 285-294.
15. Merton RK. Institutionalized patterns of evaluation in science. In: Storer NW, The Sociology of Science: Theoretical and Empirical Investigations. Chicago: University of Chicago press 1973 460-496.
16. Lamont M. Pragmatic fairness: Customary rules of deliberation. In: How professors think: Inside the curious world of academic judgment. Cambridge, MA: Harvard University Press 2009 107-158
17. Boudreau KJ, Guinan EC, Lakhani KR, Riedl C. Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance, Novelty, and Resource Allocation in Science. Management Science 2016: 1-19
18. Travis GDL, Collins HM. New light on old boys: Cognitive and institutional particularism in the peer review system. Sci., Tech., Human Values 1991 16(3):322-341.
19. Campanario JM, Acedo E. Rejecting highly cited papers: The views of scientist who encounter resistance to their discoveries from other scientists. J. Amer. Soc. Inform. Sci. Tech. 2007 58(5):734-743.
20. Anand P. Foundations of rational choice under risk. Oxford, UK: Oxford University Press 1995
21. Simon HA. Theories of bounded rationality In: McGuire CB and Radner R (Eds) Decision and organization 1972 161-176.
22. Li D. Expertise vs. bias in evaluation: Evidence from the NIH. HBS Working Paper 16-053, Harvard Business School, Boston. 2015
23. Azoulay P, Stuart T, and Wang Y Matthew: Effect or fable? Management Science 2013 60(1): 92-109.
24. Ebadi A, Schiffauerova A. How to receive more funding for your research? Get connected to the right people! PLOS ONE 2015 10(7), e0133061. doi: 10.1371/journal.pone.0133061 PMID: 26222598
25. Gillespie GW, Chubin DE, Kurzon GM. Experience with NIH peer review: researchers' cynicism and desire for change. Science, Technology, & Human Values 1985 10(3): 44-54.
26. Wold A, Wennerås C. Nepotism and sexism in peer review. Nature 1997 387(6631): 341-343. doi: 10.1038/387341a0 PMID: 9163412
27. Walker R, Barros B, Conejo R, Neumann K, Telefont M. (2015). Personal attributes of authors and reviewers, social bias and the outcomes of peer review: a case study. F1000Research 2015 4:21.
28. Cole S, Rubin L, Cole JR. Peer review in the National Science Foundation: Phase one of a study. Washington, DC: National Academy Press 1978 33-46.
29. Jayasinghe UW, Marsh HW, Bond N. A multilevel cross-classified modelling approach to peer review of grant proposals: The effects of assessor and researcher attributes on assessor ratings. Journal of the Royal Statistical Society. Series A (Statistics in Society) 2003 166(3), 279-300.
30. Marsh HW, Ball S. The interjudgmental reliability of reviews for the Journal of Educational Psychology. J. Educ. Psychol. 1981 73: 872-880.
31. Kramer M. R2 Statistics for Mixed Models. Presented at the 17th Annual Kansas State University Conference on Applied Statistics in Agriculture; 2005; April 24-26. Available: https://www.researchgate. net/profile/Matt-Kramer2/publication/266333403-R-2-STATISTICS-FOR-MIXED-MODELS/links/54d4ea160cf24647580694e9.pdf
32. Pinheiro J, Bates D, DebRoy S, Sarkar D Linear and nonlinear mixed effects models. 2007 R package version 3: 57.
33. Bornmann L, Daniel HD. The effectiveness of the peer review process: Inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition 2008 47(38), 7173-7178.
34. Gillespie A, Cornish F. Intersubjectivity: Towards a dialogical analysis. Journal for the theory of social behaviour 2010 40(1): 19-46.
35. Lee CJ. Commensuration Bias in Peer Review. Philosophy of Science 2015 82: 1272-1283.
36. Rai AK. Regulating scientific research: Intellectual property rights and the norms of science. Nw. UL Rev. 1999 94:77.
37. Bromham L, Dinnage R, Hua X. Interdisciplinary research has consistently lower funding success. Nature 2016 534:684-687. doi: 10.1038/nature18315 PMID: 27357795