The predictive validity of peer review: A selective review of the judgmental forecasting qualities of peers, and implications for innovation in science

International Journal of Forecasting

Volumn 27, Issue 1, 2011, Pages 166-182

  Benda, Wim G G ^a   Engels, Tim C E ^a,b  

^a UNIVERSITY OF ANTWERP   (Belgium)

^b ANTWERP MARITIME ACADEMY   (Belgium)

Author keywords

Advice taking; Cognitive bias; Decision making; Expert advice; Group decision making; Reliability

Indexed keywords

EID: 78149498849     PISSN: 01692070     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ijforecast.2010.03.003     Document Type: Article

References (102)

1. Abrams P.A. The predictive ability of peer review of grant proposals: the case of ecology and the US National Science Foundation. Social Studies of Science 1991, 21:111-132.
2. Armstrong J.S. Is review by peers as fair as it appears?. Interfaces 1982, 12:62-74.
3. Armstrong J.S. Research on scientific journals: implications for editors and authors. Journal of Forecasting 1982, 1:83-104.
4. Armstrong J.S. Management folklore and management science - on portfolio planning, escalation bias, and such. Interfaces 1996, 26:25-55.
5. Armstrong J.S. Peer review for journals: evidence on quality control, fairness, and innovation. Science and Engineering Ethics 1997, 3:63-84.
6. Armstrong J.S. Combining forecasts. Principles of forecasting: a handbook for researchers and practitioners 2001, 417-438. Kluwer Academic Publishers, Norwell. J.S. Armstrong (Ed.).
7. Armstrong J.S. Should the forecasting process eliminate face-to-face meetings?. Foresight: The International Journal of Applied Forecasting 2006, 3-8.
8. Armstrong J.S., Hubbard R. Does the need for agreement among reviewers inhibit the publication of controversial findings?. Behavorial and Brain Sciences 1991, 14:136-137.
9. Armstrong J.S., Pagell R. Reaping benefits from management research: lessons from the forecasting principles project. Interfaces 2003, 33:89-111.
10. Bakanic V., McPhail C., Simon R. If at first you don't succeed: review procedures for revised and resubmitted manuscripts. The American Sociologist 1990, 21:373-391.
11. Bedeian A.G. Peer review and the social construction of knowledge in the management discipline. Academy of Management Learning and Education 2004, 3:198-216.
12. Bonaccio S., Dalal R.S. Advice taking and decision-making: an integrative literature review, and implications for the organizational sciences. Organizational Behavior and Human Decision Processes 2006, 101:127-151.
13. Bornmann L., Daniel H.-D. Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees' decisions. Scientometrics 2005, 63:297-320.
14. Bornmann L., Daniel H.-D. Selecting scientific excellence through committee peer review - a citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants. Scientometrics 2006, 68:427-440.
15. Bornmann L., Daniel H.-D. Selecting manuscripts for a high-impact journal through peer review: a citation analysis of communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology 2008, 59:1841-1852.
16. Bornmann L., Daniel H.-D. What do citation counts measure? A review of studies on citing behavior. Journal of Documentation 2008, 64:45-80.
17. Bornmann L., Daniel H.-D. Extent of type I and type II errors in editorial decisions: a case study on Angewandte Chemie International Edition. Journal of Informetrics 2009, 3:348-352.
18. Bornmann L., Daniel H.-D. The manuscript reviewing process - empirical research on review requests, review sequences and decision rules in peer review. Library and Information Science Research 2010, 32:5-12.
19. Bornmann, L., Wallon, G., & Ledin, A. (2008). Does the committee peer review select the best applicants for funding? An investigation of the selection process for two European Molecular Biology Organization programmes. PLoS ONE, 3, e3480.
20. Bradley J.V. Pernicious publication practices. Bulletin of the Psychonomic Society 1981, 18:31-34.
21. Callaham M., Wears R.L., Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. The Journal of the American Medical Association 2002, 287:2847-2850.
22. Campanario J.M. Consolation for the scientist: sometimes it is hard to publish papers that are later highly-cited. Social Studies of Science 1993, 23:342-362.
23. Campanario J.M. Peer review for journals as it stands today - Part 1. Science Communication 1998, 19:181-211.
24. Campanario J.M., Acedo E. Rejecting highly cited papers: The views of scientists who encounter resistance to their discoveries from other scientists. Journal of the American Society for Information Science and Technology 2007, 58:734-743.
25. Cicchetti D.V. The reliability of peer review for manuscript and grant submissions: a cross-disciplinary investigation. Behavorial and Brain Sciences 1991, 14:119-186.
26. Daniel H.-D. Guardians of science. Fairness and reliability of peer review 1993, Wiley-VCH, Weinheim.
27. Improving psychological science. Perspectives on Psychological Science 2009, 4:1-111. (special issue).
28. Engels T.C.E., Kennedy H.P. Enhancing a Delphi study on family-focused prevention. Technological Forecasting and Social Change 2007, 74:433-451.
29. Ernst E., Saradeth T., Resch K.L. Drawbacks of peer review. Nature 1993, 363:296.
30. Fara P. Science: a four thousand year history 2009, Oxford University Press, Oxford.
31. Fiske D.W., Fogg L. But the reviewers are making different criticisms of my paper! Diversity and uniqueness in reviewer comments. American Psychologist 1990, 45:591-598.
32. Fletcher R.H., Fletcher S.W. The effectiveness of journal peer review. Peer review in health sciences 2003, 62-75. BMJ Books, London. 2nd ed. F. Godlee, T. Jefferson (Eds.).
33. Gans J.S., Shepherd G.B. How are the mighty fallen: rejected classic articles by leading economists. Journal of Economic Perspectives 1994, 8:165-179.
34. Garfield E. To be an uncited scientist is no cause for shame. Essays of an information scientist: science reviews, journalism inventiveness and other essays, Vol. 14 1991, 390-391.
35. Glidewell J.C. Reflections on thirteen years of editing AJCP. American Journal of Community Psychology 1988, 16:759-770.
36. Goodman S.N., Berlin J., Fletcher S.W., Fletcher R.H. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Annals of Internal Medicine 1994, 121:11-21.
37. Gorman G.E. The Oppenheim effect in scholarly journal publishing. Online Information Review 2007, 31:417-419.
38. Gottfredson S.D. Evaluating psychological research reports: dimensions, reliability, and correlates of quality judgments. American Psychologist 1978, 33:920-934.
39. Gould S.J. Ever since Darwin: reflections in natural history 1977, Norton, New York.
40. Graue B. The transformative power of reviewing. Educational Researcher 2006, 35:36-41.
41. Hodgson C.M. Evaluation of cardiovascular grant-in-aid applications by peer review: influence of internal and external reviewers and committees. Canadian Journal of Cardiology 1995, 11:864-868.
42. Hornbostel S., Böhmer S., Klingsporn B., Neufeld J., von Ins M. Funding of young scientist and scientific excellence. Scientometrics 2009, 79:171-190.
43. Horrobin D.F. Peer review of grant applications: a harbinger for mediocrity in clinical research?. The Lancet 1996, 348:1293-1295.
44. Jayasinghe U.W., Marsh H.W., Bond N. A new reader trial approach to peer review in funding research grants: an Australian experiment. Scientometrics 2006, 69:591-606.
45. Jefferson, T., Rudin, M., Brodney Folse, S., & Davidoff, F. (2007). Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database of Systematic Reviews, Issue 1, Art. No.: MR000016.
46. Jennings, C. G. (2006). Quality and value: the true purpose of peer review. Retrieved 13.7.09. http://www.nature.com/nature/peerreview/debate/nature05032.html.
47. Judge T.A., Cable D.M., Colbert A.E., Rynes S.L. What causes a management article to be cited - article, author, or journal?. Academy of Management Journal 2007, 50:491-506.
48. Kameda T., Sugimori S. Procedural influence in 2-step group decision-making - power of local majorities in consensus formation. Journal of Personality and Social Psychology 1995, 69:865-876.
49. Kerr N.L., MacCoun R.J., Kramer G.P. Bias in judgment: comparing individuals and groups. Psychological Review 1996, 103:687-719.
50. Kerr N.L., Tindale R.S. Group performance and decision making. Annual Review of Psychology 2004, 55:623-655.
51. Kostoff R.N. Federal research impact assessment: axioms, approaches, applications. Scientometrics 1995, 34:163-206.
52. Kupfersmid J., Wonderly D.M. An author's guide to publishing better articles in better journals in the behavioural sciences 1994, Wiley & Sons, Hoboken, NJ.
53. Laband D.N. Is there value-added from the review process in economics? Preliminary evidence from authors. The Quarterly Journal of Economics 1990, 105:341-352.
54. Langfeldt L. The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science 2001, 31:820-841.
55. Langfeldt L. Expert panels evaluating research: decision-making and sources of bias. Research Evaluation 2004, 13:51-62.
56. Langfeldt L. The policy challenges of peer review: managing bias, conflict of interests and interdisciplinary assessments. Research Evaluation 2006, 15:31-41.
57. Lawrence M., Goodwin P., O'Conner M., Önkal D. Judgmental forecasting: a review of progress over the last 25 years. International Journal of Forecasting 2006, 22:493-518.
58. Lawrence P.A. The politics of publication. Nature 2003, 422:259-261.
59. Lehmann S., Jackson A.D., Lautrup B.E. Measures for measures. Nature 2006, 444:1003-1004.
60. Leimu R., Koricheva J. What determines the citation frequency of ecological papers?. Trends in Ecology and Evolution 2005, 20:28-32.
61. Levin I.P., Prosansky C.M., Heller D., Brunick B.M. Prescreening of choice options in 'positive' and 'negative' decision-making tasks. Journal of Behavioral Decision Making 2001, 14:279-293.
62. Levitt, S. D., & List, J. A. (2009). Was there really a Hawthorne effect at the Hawthorne plant? An analysis of the original illumination experiments. NBER Working Papers Series: Working Paper 15016. Retrieved 1.07.09. http://www.nber.org/papers/w15016.pdf.
63. Mahoney M.J. Publication prejudices: an experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research 1977, 1:161-175.
64. Marsh H.W., Bornmann L. Do women have less success in peer review?. Nature 2009, 459:602.
65. Marsh H.W., Jayasinghe U.W., Bond N. Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability. American Psychologist 2008, 63:160-168.
66. Mavis B., Katz M. Evaluation of a program supporting scholarly productivity for new investigators. Academic Medicine 2003, 78:757-765.
67. McDonals R.J., Cloft H.J., Kallmes D.F. Fate of manuscripts previously rejected by the American Journal of Neuroradiology: a follow-up analysis. American Journal of Neuroradiology 2009, 30:253-256.
68. Melin G., Danell R. The top eight percent: development of approved and rejected applicants for a prestigious grant in Sweden. Science and Public Policy 2006, 33:702-712.
69. Moed H.F. The future of research evaluation rests with an intelligent combination of advanced metrics and transparent peer review. Science and Public Policy 2007, 34:575-584.
70. Nederhof A., van Raan A. Peer review and bibliometric indicators of scientific performance: a comparison of cum laude doctorates with ordinary doctorates in physics. Scientometrics 1987, 11:333-350.
71. Nederhof A., van Raan A. A validation study of bibliometric indicators: the comparative performance of cum laude doctorates in chemistry. Scientometrics 1989, 17:427-435.
72. Nickerson R.S. What authors want from journal reviewers and editors. American Psychologist 2005, 60:661-662.
73. Obrecht M., Tibelius K., D'Aloisio G. Examining the value added by committee discussion in the review of applications for research awards. Research Evaluation 2007, 16:79-91.
74. Opthof T., Furstner F., van Geer M., Coronel R. Regrets or no regrets? No regrets! The fate of rejected manuscripts. Cardiovascular Research 2000, 45:255-258.
75. Peters D.P., Ceci S.J. Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences 1982, 5:187-195.
76. Pierie J.P.E., Walvoort H.C., Overbeke A.J.P. Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde. The Lancet 1996, 348:1480-1483.
77. Racki G. Rank-normalized journal impact factor as a predictive tool. Archivum Immunologiae et Therapiae Experimentalis 2009, 57:39-43.
78. Rowe G., Wright G. Expert opinions in forecasting: the role of the Delphi technique. Principles of forecasting: a handbook for researchers and practitioners 2001, 125-144. Kluwer Academic Publishers, Norwell. J.S. Armstrong (Ed.).
79. Rowe G., Wright G., McColl A. Judgment change during Delphi-like procedures: the role of majority influence, expertise, and confidence. Technological Forecasting and Social Change 2005, 72:377-399.
80. Samuelson W., Zeckhauser R. Status quo bias in decision making. Journal of Risk and Uncertainty 1988, 1:7-59.
81. Saper C.B., Maunsell J.H.R. The neuroscience peer review consortium. The Journal of Comparative Neurology 2009, 513:333-334.
82. Seglen P.O. Causal relationship between article citedness and journal impact. Journal of the American Society for Information Science 1994, 45:1-11.
83. Shafir E. Choosing versus rejecting: Why some options are both better and worse than others. Memory and Cognition 1993, 21:546-556.
84. Siegelman S.S. Assassins and Zealots: variations in peer review. Radiology 1991, 178:637-642.
85. Skoda, Z. (6-11-2008). Re: The kind of email I don't need. Retrieved 22.07.09. http://golem.ph.utexas.edu/category/2008/11/the_kind_of_email_i_dont_need.html#c019806.
86. Sniezek J.A., Buckley T. Cueing and cognitive conflict in judge-advisor decision making. Organizational Behavior and Human Decision Processes 1995, 62:159-174.
87. Soll J.B., Larrick R.P. Strategies for revising judgment: How (and how well) people use others' opinions. Journal of Experimental Psychology - Learning Memory and Cognition 2009, 35:780-805.
88. Starbuck W.H. How much better are the most prestigious journals? The statistics of academic publication. Organization Science 2005, 16:180-200.
89. Straub D.W. Type II reviewing errors and the search for exciting papers. MIS Quarterly 2008, 32:5-10.
90. Suls J., Martin R. The air we breathe: A critical look at practices and alternatives in the peer-review process. Perspectives on Psychological Science 2009, 4:40-50.
91. Sweitzer B.J., Cullen D.J. How well does a journal's peer review process function? A survey of authors' opinions. The Journal of the American Medical Association 1994, 272:152-153.
92. Thornley R., Spence M.W., Taylor M., Magnan J. New decision tool to evaluate award selection process. The Journal of Research Administration 2002, 33:49-56.
93. Travis G.D.L., Collins H.M. New light on old boys: cognitive and institutional particularism in the peer review system. Science, Technology and Human Values 1991, 16:322-341.
94. Tversky A., Kahneman D. Loss aversion in riskless choice: a reference-dependent model. The Quarterly Journal of Economics 1991, 107:1039-1061.
95. van den Besselaar P, Leydesdorff L. Past performance, peer review, and project selection: a case study in the social and behavorial sciences. Research Evaluation 2009, 18:273-288.
96. van Raan A.F.J. Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises. Scientometrics 1996, 36:397-420.
97. van Raan A.F.J. Measuring science. Capita selecta of current main issues. Handbook of quantitative science and technology research. The use of publication and patent statistics in studies of S&T systems 2004, 19-50. Kluwer Academic Publishers, Dordrecht. H.F. Moed, W. Glänzel, U. Schmoch (Eds.).
98. van Raan A. Sleeping Beauties in science. Scientometrics 2004, 59:467-472.
99. Wakin M., Rozell C., Davenport M., Laska J. Letter from the editors. Rejecta Mathematica 2009, 1:1-3.
100. Weller A.C. Editorial peer review: its strengths and weaknesses 2001, American Society for Information Science and Technology, Medford, NJ.
101. Wilson J.D. Peer review and publication: Presidential address before the 70th annual meeting of the American Society for Clinical Investigation. Journal of Clinical Investigation 1978, 61:1697-1701.
102. Wright G., Lawrence M.J., Collopy F. The role and validity of judgment in forecasting. International Journal of Forecasting 1996, 12:1-8.