Examining the value added by committee discussion in the review of applications for research awards

Research Evaluation

Volumn 16, Issue 2, 2007, Pages 79-91

  Obrecht, Michael ^a   Tibelius, Karl ^a   D'Aloisio, Guy ^a,b  

^a CANADIAN INSTITUTES OF HEALTH RESEARCH   (Canada)

^b Genome Canada   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 34548600110     PISSN: 09582029     EISSN: None     Source Type: Journal    
DOI: 10.3152/095820207X223785     Document Type: Article

References (47)

1. Abrams, P A 1991. The predictive ability of peer review of grant proposals: the case of ecology and the National Science Foundation. Social Studies of Science, 21, 111-132.
2. Agnew, B 1999. NIH eyes sweeping reform of peer review. Science, 286, 1074-1076, 5 November.
3. Aguilar, A, T Ingemansson, A Hogan and E Magnien 1998. Peer review of proposals in the biotechnology programme of the European Union. Research Evaluation, 7(3), 141-146.
4. Barr, M 2002. Grantsmanship helps no one. University Affairs (Association of Universities and Colleges of Canada), 5, March.
5. Berezin, A 1998. The perils of centralized research funding systems. Knowledge, Technology and Policy, 11(3), 5-26.
6. Berkowitz, P 1998. Research gets a new scorecard. University Affairs (Association of Universities and Colleges of Canada), 19-20, April.
7. Birkett, J 1994. The review process for applied-research grant proposals: suggestion for revision. Canadian Medical Association Journal, 150(8), 1227-1229.
8. Bornmann, L and H D Daniel 2005. Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees' decisions. Scientometrics, 63(2), 297-320.
9. Chubin, D E 1994. Grants peer review in theory and practice. Evaluation Review, 18(1), 20-30.
10. Cicchetti, D V 1991. The reliability of peer review for manuscript and grant submissions: a cross-disciplinary investigation. Behavioral and Brain Sciences, 14, 119-186.
11. CIHR, Canadian Institutes of Health Research 2003. A CIHR Process for Awarding Fellowships: Documents to Guide the Development of a Valid, Reliable and Cost-Effective Process. Ottawa: CIHR.
12. CIHR, Canadian Institutes of Health Research 2005. Fellowships Selection Process Improvement Project: Technical Reports. Ottawa: CIHR.
13. Claveria, L E, E Guallar, J Cami, J Conde, R Pastor, J R Ricoy, E Rodriguez-Farre, F Ruiz-Palomo and E Munoz 2000. Does peer review predict the performance of research projects in health sciences? Scientometrics, 47(1), 11-23.
14. Cole, S, J R Cole and G S Simon 1981. Chance and consensus in peer review. Science, 214, 881-886, 20 November.
15. Cole, S, L Rubin and J R Cole 1977. Peer review and the support of science. Scientific American, 237(4), 34-41.
16. Cook, W D, B Golany, M Kress, M Penn and T Raviv 2005. Optimal allocation of proposals to reviewers to facilitate effective ranking. Management Science, 51(4), 655-661.
17. Davis, J H 1992. Some compelling intuitions about group consensus decision, theoretical and empirical research, and interpersonal aggregation phenomena: selected examples, 1950-1990. Organizational Behavior and Human Decision Processes, 52, 3-38.
18. Dawes, R M, D Faust and P E Meehl 1989. Clinical versus actuarial judgement. Science, 243, 1668-1674.
19. Fliesler, S J 1997. Rethinking grant peer review. Science, 275, 1399, 7 March.
20. Gillett, R 1993. Is medical research well served by peer review? British Medical Journal, 306, 1672-1675.
21. Glantz, S A and L A Bero 1994. Inappropriate and appropriate selection of 'peers' in grant review. Journal of the American Medical Association, 272(2), 114-116.
22. Green, J G, F Calhoun, I Nierzwicki, J Brackett and P Meier 1989. Rating intervals: an experiment in peer review. FASEB Journal, 3, 1987-1992.
23. Hodgson, C 1995. Evaluation of cardiovascular grant-in-aid applications by peer review: influence of internal and external reviewers and committees. Canadian Journal of Cardiology, 11, 864-868.
24. Hodgson, C 1997. How reliable is peer review? A comparison of operating grant proposals simultaneously submitted to two similar, peer review systems. Journal of Clinical Epidemiology, 50, 1189-1195.
25. Horrobin, D 1996. Peer review of grant applications: a harbinger for mediocrity in clinical research? Lancet, 348, 1293-1295, 9 November.
26. Kerr, N L, R J MacCoun and G P Kramer 1996. Bias in judgement: comparing individuals and groups. Psychological Review, 103(4), 687-719.
27. Klahr, D 1985. Insiders, outsiders and efficiency in a National Science Foundation panel. American Psychologist, 40, 148-154.
28. Langfeldt, L 2001. The decision-making constraints and processes of peer review, and their effects on the review outcome. Social Studies of Science, 31(6), 820-841.
29. Langfeldt, L 2004. Expert panels evaluating research: decision-making and sources of bias. Research Evaluation, 13(1), 51-62.
30. Levy, J A 1984. Peer review: the continual need for reassessment. Cancer Investigation, 2(4), 311-320.
31. Moxham, H and J Anderson 1992. Peer review: a view from the inside. Science and Technology Policy, 7-15, February.
32. NIH, National Institutes of Health 1996. Report of the Committee on Rating of Grant Applications. Bethesda: NIH. 〈http://grants.nih.gov/ grants/peer/rga.txt〉, last accessed 11 July 2007.
33. O'Brecht, M, R Pihl and P Bois 1989. Criteria for granting research training awards to graduate students. Research in Higher Education, 30(6), 647-664.
34. Obrecht, M, S Robertson and J Hood 1998. Study of the System for Selecting Recipients of MRC Doctoral Research Awards. Ottawa: Medical Research Council.
35. Pagano, M 2006. American Idol and NIH review. Cell, 126, 637-638, 25 August.
36. Roy, R 1985. Funding science: the real defects of peer review and an alternative to it. Science, Technology, and Human Values, 10(3), 73-81.
37. Seiken, J 1992. A reviewer's-eye view of evaluation processes at NIH and NSF. Scientist, 19-20, March.
38. Smith, R 1988. Glimpses of the National Institutes of Health Research II: review systems and evaluation. British Medical Journal, 296, 691-695, 5 March.
39. Surowiecki, J 2004. The Wisdom of Crowds: Why the Many Are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. New York: Random House/Doubleday.
40. Swift, M 1996. Innovative research and NIH grant review. Journal of NIH Research, 8, 18-20.
41. Thorngate, W, N Faregh and M Young 2002. Mining the Archives: Analyses of CIHR Research Grant Adjudications. A report to the Canadian Institutes of Health Research. Ottawa: CIHR. 〈http://http-server.carleton.ca/~warrent/ reports〉, last accessed 11 July 2007.
42. Thornley, R, M W Spence, M Taylor and J Magnan 2002. New decision tool to evaluate award selection process. Journal of Research Administration, 33(2), 49-56.
43. Travis, GDL and H M Collins 1991. New light on old boys: cognitive and institutional particularism in the peer review system. Science, Technology and Human Values, 16(3), 322-341.
44. Vener, K J, E J Feuer and L Gorelic 1993. A statistical model validating triage for the peer review process: keeping the competitive applications in the review pipeline. FASEB Journal, 7, 1312-1319.
45. Wenneras, C and A Wold 1997. Nepotism and sexism in peer review. Nature, 387, 341-343, 22 May.
46. Wessely, S 1998. Peer review of grant applications: what do we know? Lancet, 352, 301-305, 25 July.
47. Wessely, S and F Wood 1999. Peer review of grant applications: a systematic review. In Peer Review in Health Sciences, eds. F Godlee and T Jefferson. London: BMJ Books.