Scientific peer review

Annual Review of Information Science and Technology

Volumn 45, Issue , 2011, Pages 197-245

  Bornmann, Lutz ^a  

^a ETH ZURICH   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84877103582     PISSN: 00664200     EISSN: None     Source Type: Book Series    
DOI: 10.1002/aris.2011.1440450112     Document Type: Article

References (259)

1. Abelson, P. H. (1980). Scientific communication. Science, 209(4452), 60-62.
2. Abramowitz, S. I., Gomes, B., & Abramowitz, C. V. (1975). Publish or politic: Referee bias in manuscript review. Journal of Applied Social Psychology, 3(5), 187-200.
3. Abrams, P. A. (1991). The predictive ability of peer review of grant proposals: The case of ecology and the United States National Science Foundation. Social Studies of Science, 21(1), 111-132.
4. Alberts, B., Hanson, B., & Kelner, K. L. (2008). Reviewing peer review. Science, 321(5885), 15.
5. Armstrong, A. W., Idriss, S. Z., Kimball, A. B., & Bernhard, J. D. (2008). Fate of manuscripts declined by the Journal of the American Academy of Dermatology. Journal of the American Academy of Dermatology, 58(4), 632-635.
6. Armstrong, J. S. (1997). Peer review for journals: Evidence on quality control, fairness, and innovation. Science and Engineering Ethics, 3(1), 63-84.
7. Armstrong, P. W., Caverson, M. M., Adams, L., Taylor, M., & Olley, P. M. (1997). Evaluation of the Heart and Stroke Foundation of Canada Research Scholarship Program: Research productivity and impact. Canadian Journal of Cardiology, 13(5), 507-516.
8. Bailar, J. C. (1991). Reliability, fairness, objectivity, and other inappropriate goals in peer review. Behavioral and Brain Sciences, 14(1), 137-138.
9. Bakanic, V., McPhail, C., & Simon, R. J. (1987). The manuscript review and decision-making process. American Sociological Review, 52(5), 631-642.
10. Bakanic, V., McPhail, C., & Simon, R. J. (1989). Mixed messages: Referees' comments on the manuscripts they review. Sociological Quarterly, 30(4), 639-654.
11. Bartley, W. W. (1984). The retreat to commitment (2nd ed.). La Salle, IL: Open Court.
12. Baxt, W. G., Waeckerle, J. F., Berlin, J. A., & Callaham, M. L. (1998). Who reviews the reviewers Feasibility of using a fictitious manuscript to evaluate peer reviewer performance. Annals of Emergency Medicine, 32(3), 310-317.
13. Bedeian, A. G. (2003). The manuscript review process: The proper roles of authors, referees, and editors. Journal of Management Inquiry, 12(4), 331-338.
14. Bedeian, A. G. (2004). Peer review and the social construction of knowledge in the management discipline. Academy of Management Learning and Education, 3(2), 198-216.
15. Bensman, S. J. (2007). Garfield and the impact factor. Annual Review of Information Science and Technology, 41, 93-155.
16. Bernard, H. R. (1980). Report from the editor. Human Organization, 39(4), 366-369.
17. Bertout, C., & Schneider, P. (2004). Editorship and peer-review at A&A. Astronomy & Astrophysics, 420, E1-E14.
18. Bhandari, M., Templeman, D., & Tornetta, P. (2004). Interrater reliability in grading abstracts for the Orthopaedic Trauma Association. Clinical Orthopaedics and Related Research, 423, 217-221.
19. Biagioli, M. (2002). From book censorship to academic peer review. Emergences, 12(1), 11-45.
20. Blank, R. M. (1991). The effects of double-blind versus single-blind reviewing: Experimental evidence from the American Economic Review. American Economic Review, 81(5), 1041-1067.
21. Bohannon, R. W. (1986). Agreement among reviewers. Physical Therapy, 66(9), 1431-1432.
22. Borgman, C. L. (2007). Scholarship in the digital age: Information, infrastructure, and the internet. Cambridge, MA: MIT Press.
23. Borgman, C. L., & Furner, J. (2002). Scholarly communication and bibliometrics. Annual Review of Information Science and Technology, 36, 3-72.
24. Bornmann, L., & Daniel, H.D. (2005a). Committee peer review at an international research foundation: Predictive validity and fairness of selection decisions on post-graduate fellowship applications. Research Evaluation, 14(1), 15-20.
25. Bornmann, L., & Daniel, H.-D. (2005b). Criteria used by a peer review committee for selection of research fellows: A Boolean probit analysis. International Journal of Selection and Assessment, 13(4), 296-303.
26. Bornmann, L., & Daniel, H.-D. (2005c). Selection of research fellowship recipients by committee peer review: Analysis of reliability, fairness and predictive validity of Board of Trustees' decisions. Scientometrics, 63(2), 297-320.
27. Bornmann, L., & Daniel, H.-D. (2006). Selecting scientific excellence through committee peer review: A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants. Scientometrics, 68(3), 427-440.
28. Bornmann, L., & Daniel, H.-D. (2007a). Convergent validation of peer review decisions using the h index: Extent of and reasons for type I and type II errors. Journal of Informetrics, 1(3), 204-213.
29. Bornmann, L., & Daniel, H.-D. (2007b). Gatekeepers of science: Effects of external reviewers' attributes on the assessments of fellowship applications. Journal of Informetrics, 1(1), 83-91.
30. Bornmann, L., & Daniel, H.-D. (2008a). The effectiveness of the peer review process: Interreferee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition, 47(38), 7173-7178.
31. Bornmann, L., & Daniel, H.-D. (2008b). Selecting manuscripts for a high impact journal through peer review: A citation analysis of communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology, 59(11), 1841-1852.
32. Bornmann, L., & Daniel, H.-D. (2009). The luck of the referee draw: The effect of exchanging reviews. Learned Publishing, 22(2), 117-125.
33. Bornmann, L., & Daniel, H. D. (in press). The manuscript reviewing process: Empirical research on review requests, review sequences and decision rules in peer review. Library & Information Science Research.
34. Bornmann, L., Mutz, R., & Daniel, H.-D. (2007a). Gender differences in grant peer review: A meta-analysis. Journal of Informetrics, 1(3), 226-238.
35. Bornmann, L., Mutz, R., & Daniel, H.-D. (2007b). Row-column (RC) association model applied to grant peer review. Scientometrics, 73(2), 139-147.
36. Bornmann, L., Mutz, R., & Daniel, H.-D. (2008). How to detect indications of potential sources of bias in peer review: A generalized latent variable modeling approach exemplified by a gender study. Journal of Informetrics, 2(4), 280-287.
37. Bornmann, L., Mutz, R., & Daniel, H.-D. (2009). A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. (Submitted for publication)
38. Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H.-D. (2008). Use of citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics, 8, 93-102.
39. Bornmann, L., Nast, I., & Daniel, H.-D. (2008). Do editors and referees look for signs of scientific misconduct when reviewing manuscripts A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics, 77(3), 415-432.
40. Bornmann, L., Wallon, G., & Ledin, A. (2008). Does the committee peer review select the best applicants for funding An investigation of the selection process for two European Molecular Biology Organization programmes. PLoS One, 3(10), e3480.
41. Bornstein, R. F. (1991). The predictive validity of peer-review: A neglected issue. Behavioral and Brain Sciences, 14(1), 138-139.
42. Brennan, R. L., & Prediger, D. J. (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3), 687-699.
43. British Academy (2007). Peer review: The challenges for the humanities and social sciences. London: The Academy.
44. Brody, T., Harnad, S., & Carr, L. (2006). Earlier web usage statistics as predictors of later citation impact. Journal of the American Society for Information Science and Technology, 57(8), 1060-1072.
45. Caelleigh, A. S., Hojat, M., Steinecke, A., & Gonnella, J. S. (2003). Effects of reviewers' gender on assessments of a gender-related standardized manuscript. Teaching and Learning in Medicine, 15(3), 163-167.
46. Campanario, J. M. (1998a). Peer review for journals as it stands today: Part 1. Science Communication, 19(3), 181-211.
47. Campanario, J. M. (1998b). Peer review for journals as it stands today: Part 2. Science Communication, 19(4), 277-306.
48. Carter, G. (1982). What we know and do not know about the peer review system (Rand Report N-1878-RC/NIH). Santa Monica, CA: RAND Corporation.
49. Carter, G., Cooper, W., Lai, C., & March, D. (1978). The consequences of unfunded NIH applications for the investigator and his research (Rand Report R-2229-NIH). Santa Monica, CA: RAND Corporation.
50. Chapman, G. B., & McCauley, C. (1994). Predictive validity of quality ratings of National Science Foundation graduate fellows. Educational and Psychological Measurement, 54(2), 428-438.
51. Chew, F. S. (1991). Fate of manuscripts rejected for publication in the AJR. American Journal of Roentgenology, 156(3), 627-632.
52. Chubin, D., & Hackett, E. (1990). Peerless science: Peer review and U.S. science policy. Albany, NY: State University of New York Press.
53. Chubin, D. E. (1982). Reforming peer-review: From recycling to reflexivity. Behavioral and Brain Sciences, 5(2), 204.
54. Cicchetti, D. V. (1991). The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14(1), 119-135.
55. Cicchetti, D. V. (1997). Referees, editors, and publication practices: Improving the reliability and usefulness of the peer review system. Science and Engineering Ethics, 3(1), 51-62.
56. Cicchetti, D. V. (1999). Guardians of science: Fairness and reliability of peer review. Journal of Clinical and Experimental Neuropsychology, 21(3), 412-421.
57. Cicchetti, D. V., & Conn, H. O. (1976). A statistical analysis of reviewer agreement and bias in evaluating medical abstracts. Yale Journal of Biology and Medicine, 49(4), 373-383.
58. Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low Kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6), 551-558.
59. Cole, J. R. (1979). Fair science: Women in the scientific community. New York: The Free Press.
60. Cole, J. R. (2000). The role of journals in the growth of scientific knowledge. In B. Cronin & H. B. Atkins (Eds.), The web of knowledge: A Festschrift in honor of Eugene Garfield (pp. 109-142). Medford, NJ: Information Today, Inc.
61. Cole, S. (1992). Making science. Between nature and society. Cambridge, MA: Harvard University Press.
62. Cole, S. (2004). Merton's contribution to the sociology of science. Social Studies of Science, 34(6), 829-844.
63. Cole, S., & Cole, J. R. (1967). Scientific output and recognition: A study in operation of reward system in science. American Sociological Review, 32(3), 377-390.
64. Cole, S., Cole, J. R., & Simon, G. A. (1981). Chance and consensus in peer-review. Science, 214(4523), 881-886.
65. Cole, S., & Fiorentine, R. (1991). Discrimination against women in science: The confusion of outcome with process. In H. Zuckerman, J. R. Cole, & J. T. Bruer (Eds.), The outer circle: Women in the scientific community (pp. 205-226). London: W. W. Norton.
66. Cronin, B. (2005). The hand of science: Academic writing and its rewards. Lanham, MD: Scarecrow Press.
67. Cronin, B., & McKenzie, G. (1992). The trajectory of rejection. Journal of Documentation, 48(3), 310-317.
68. Daniel, H.-D. (1993/2004). Guardians of science: Fairness and reliability of peer review. Weinheim, Germany: Wiley-VCH. Retrieved July 16, 2004, from DOI: 10.1002/3527602208
69. Davo, M. D., Vives, C., & Alvarez-Dardet, C. (2003). Why are women underused in the JECH peer review process Journal of Epidemiology and Community Health, 57(12), 936-937.
70. Demicheli, V., & Pietrantonj, C. (2007). Peer review for improving the quality of grant applications. Cochrane Database of Systematic Reviews, Issue 2. Art. No.: MR000003. DOI: 10.1002/14651858.MR000003.pub2
71. Duncan, G. J., & Magnuson, K. A. (2003). The promise of random-assignment social experiments for understanding well-being and behavior. Current Sociology, 51(5), 529-541.
72. Eckberg, D. L. (1991). When nonreliability of reviews indicates solid science. Behavioral and Brain Sciences, 14(1), 145-146.
73. Eckes, T. (2004). Rater agreement and rater severity: A many-faceted Rasch analysis of performance assessments in the "Test Deutsch als Fremdsprache" (TestDaF). Diagnostica, 50(2), 65-77.
74. Editor to quit over hoax open-access paper. (2009). Nature, 459, 901.
75. Efron, B., & Gong, G. (1983). A leisurely look at the bootstrap, the jackknife, and cross-validation. American Statistician, 37(1), 36-48.
76. Ehses, I. (2004). By scientists, for scientists. The Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) and how it functions. B.I.F. Futura, 19, 170-177.
77. Eisenhart, M. (2002). The paradox of peer review: Admitting too much or allowing too little Research in Science Education, 32(2), 241-255.
78. Epstein, W. M. (1990). Confirmational response bias among social-work journals. Science, Technology, & Human Values, 15(1), 9-38.
79. Ernst, E., Saradeth, T., & Resch, K. L. (1993). Drawbacks of peer review. Nature, 363(6427), 296.
80. Evidence Ltd. (2007). The use of bibliometrics to measure research quality in UK higher education institutions. London: Universities UK.
81. Eysenck, H. J., & Eysenck, S. B. G. (1992). Peer review: Advice to referees and contributors. Personality and Individual Differences, 13(4), 393-399.
82. Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low Kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543-549.
83. Feist, G. J. (2006). The psychology of science and the origins of the scientific mind. New Haven, CT: Yale University Press.
84. Ferber, M. A., & Teiman, M. (1980). Are women economists at a disadvantage in publishing journal articles Eastern Economic Journal, 6(3-4), 189-193.
85. Figueredo, E. (2006). The numerical equivalence between the impact factor of journals and the quality of the articles. Journal of the American Society for Information Science and Technology, 57(11), 1561.
86. Fiske, D. W., & Fogg, L. (1990). But the reviewers are making different criticisms of my paper: Diversity and uniqueness in reviewer comments. American Psychologist, 45(5), 591-598.
87. Fleiss, J. (1981). Statistical methods for rates and proportions. New York: Wiley VCH.
88. Fleiss, J. L. (1982). Deception in the study of the peer-review process. Behavioral and Brain Sciences, 5(2), 210-211.
89. Fogg, L., & Fiske, D. W. (1993). Foretelling the judgments of reviewers and editors. American Psychologist, 48(3), 293-294.
90. Fox, T. (1965). Crisis in communication: The functions and future of medical publication. London: Athlone Press.
91. Frey, B. S. (2003). Publishing as prostitution Choosing between one's own ideas and academic success. Public Choice, 116(1-2), 205-223.
92. Fuhrer, M. J., & Grabois, M. (1985). Grant application and review procedures of the National Institute of Handicapped Research: Survey of applicant and peer reviewer opinions. Archives of Physical Medicine and Rehabilitation, 66(5), 318-321.
93. Gans, J. S., & Shepherd, G. B. (1994). How are the mighty fallen: Rejected classic articles by leading economists. Journal of Economic Perspectives, 8(1), 165-179.
94. Garfield, E. (2004). Historiographic mapping of knowledge domains literature. Journal of Information Science, 30(2), 119-145.
95. Garfunkel, J. M., Ulshen, M. H., Hamrick, H. J., & Lawson, E. E. (1994). Effect of institutional prestige on reviewers' recommendations and editorial decisions. Journal of the American Medical Association, 272(2), 137-138.
96. Geisler, E. (2000). The metrics of science and technology. Westport, CT: Quorum Books.
97. Geisler, E. (2001). The mires of research evaluation. The Scientist, 15(10), 39.
98. Gibson, M., Spong, C. Y., Simonsen, S. E., Martin, S., & Scott, J. R. (2008). Author perception of peer review. Obstetrics and Gynecology, 112(3), 646-651.
99. Gilbert, J. R., Williams, E. S., & Lundberg, G. D. (1994). Is there gender bias in JAMA's peer review process Journal of the American Medical Association, 272(2), 139-142.
100. Gillespie, G. W., Chubin, D. E., & Kurzon, G. M. (1985). Experience with NIH peer review: Researchers' cynicism and desire for change. Science, Technology, & Human Values, 52, 44-54.
101. Glass, G. V. (1976). Primary, secondary, and meta-analysis. Educational Researcher, 5, 3-8.
102. Godlee, F., & Dickersin, K. (2003). Bias, subjectivity, chance, and conflict of interest. In F. Godlee & J. Jefferson (Eds.), Peer review in health sciences (2nd ed., pp. 91-117). London: BMJ Publishing Group.
103. Goldberg, P. (1968). Are women prejudiced against women Transactions, 5(5), 28-30.
104. Gölitz, P. (2008). Appeals. Angewandte Chemie International Edition, 47(38), 7144-7145.
105. Goodman, L. A. (1984). The analysis of cross-classified data having ordered categories. Cambridge, MA: Harvard University Press.
106. Goodman, S. N., Berlin, J., Fletcher, S. W., & Fletcher, R. H. (1994). Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Annals of Internal Medicine, 121(1), 11-21.
107. Gordon, M. D. (1984). How authors select journals: A test of the reward maximization model of submission behavior. Social Studies of Science, 14(1), 27-43.
108. Gorman, G. E. (2007). The Oppenheim effect in scholarly journal publishing. Online Information Review, 31(4), 417-419.
109. Gosden, H. (2003). "Why not give us the full story": Functions of referees' comments in peer reviews of scientific research papers. Journal of English for Academic Purposes, 2(2), 87-101.
110. Grayson, L. (2002). Evidence based policy and the quality of evidence: Rethinking peer review. London: Centre for Evidence Based Policy and Practice (ESRC).
111. Guston, D. H. (2003). The expanding role of peer review processes in the United States. In P. Shapira & S. Kuhlmann (Eds.), Learning from science and technology policy evaluation: Experiences from the United States and Europe (pp. 81-97). Cheltenham, UK: Edward Elgar.
112. Hackett, E. J., & Chubin, D. E. (2003, February). Peer review for the 21st century: Applications to education research. Paper presented at the conference Peer Review of Education Research Grant Applications: Implications, Considerations, and Future Directions, Washington, DC.
113. Hagstrom, W. O. (1965). The scientific community. New York: Basic Books.
114. Hames, I. (2007). Peer review and manuscript management of scientific journals: Guidelines for good practice. Oxford, UK: Blackwell.
115. Hansson, F. (2002). How to evaluate and select new scientific knowledge Taking the social dimension seriously in the evaluation of research quality. Vest, 15(2-3), 27-52.
116. Hargens, L. L., & Herting, J. R. (1990). A new approach to referees assessments of manuscripts. Social Science Research, 19(1), 1-16.
117. Harnad, S. (1990). Scholarly skywriting and the prepublication continuum of scientific inquiry. Psychological Science, 1(6), 342-344.
118. Harnad, S. (1996). Implementing peer review on the net: Scientific quality control in scholarly electronic journals. In P. R. Peek & G. B. Newby (Eds.), Scholarly publishing: The electronic frontier (pp. 103-118). Cambridge, MA: MIT Press.
119. Harnad, S. (2008). Validating research performance metrics against peer rankings. Ethics in Science and Environmental Politics, 8, 103-107.
120. Hemlin, S. (1996). Research on research evaluations. Social Epistemology, 10(2), 209-250.
121. Hemlin, S., & Rasmussen, S. B. (2006). The shift in academic quality control. Science, Technology, & Human Values, 31(2), 173-198.
122. Hodgson, C. (1997). How reliable is peer review An examination of operating grant proposals simultaneously submitted to two similar peer review systems. Journal of Clinical Epidemiology, 50(11), 1189-1195.
123. Hoffmann, H., Joye, D., Kuhn, F., & Métral, G. (2002). Der SNF im Spiegel der Forschenden: Synthesebericht. Neuchâtel, Switzerland: Schweizerischer Informations und Datenarchivdienst für die Sozialwissenschaften (SIDOS).
124. Hojat, M., Gonnella, J. S., & Caelleigh, A. S. (2003). Impartial judgment by the "gatekeepers" of science: Fallibility and accountability in the peer review process. Advances in Health Sciences Education, 8(1), 75-96.
125. Honig, W. M. (1982). Peer review in the physical sciences: An editor's view. Behavioral and Brain Sciences, 5(2), 216-217.
126. Hornbostel, S. (1997). Wissenschaftsindikatoren: Bewertungen in der Wissenschaft. Opladen, Germany: Westdeutscher Verlag.
127. Hornbostel, S., Böhmer, S., Klingsporn, B., Neufeld, J., & von Ins, M. (2009). Funding of young scientist and scientific excellence. Scientometrics, 79(1), 171-190.
128. Hornbostel, S., & Olbrecht, M. (2007). Peer Review in der DFG: Die Fachkollegiaten (iFQWorking Paper No. 2). Bonn, Germany: Institut für Forschungsinformation und Qualitätssicherung.
129. Ingelfinger, F. J. (1974). Peer review in biomedical publication. American Journal of Medicine, 56(5), 686-692.
130. Jayasinghe, U. W. (2003). Peer review in the assessment and funding of research by the Australian Research Council. Greater Western Sydney, Australia: University of Western Sydney.
131. Jayasinghe, U. W., Marsh, H. W., & Bond, N. (2001). Peer review in the funding of research in higher education: The Australian experience. Educational Evaluation and Policy Analysis, 23(4), 343-364.
132. Jayasinghe, U. W., Marsh, H. W., & Bond, N. (2003). A multilevel cross-classified modelling approach to peer review of grant proposals: The effects of assessor and researcher attributes on assessor ratings. Journal of the Royal Statistical Society Series a-Statistics in Society, 166, 279-300.
133. Jennings, C. G. (2006). Quality and value: The true purpose of peer review. What you can't measure, you can't manage: The need for quantitative indicators in peer review. Nature. Retrieved July 6, 2006, from www.nature.com/nature/peerreview/debate/nature05032. html
134. Johnson, V. E. (2008). Statistical analysis of the National Institutes of Health peer review system. Proceedings of the National Academy of Sciences, 105(32), 11076-11080.
135. Joint Committee on Quantitative Assessment of Research (2008). Citation statistics. A report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS). Berlin, Germany: International Mathematical Union (IMU).
136. Journal of the American Medical Association. (1990). Guarding the guardians: Research on editorial peer review. Selected proceedings from the First International Congress on Peer Review in Biomedical Publication. Journal of the American Medical Association, 263(10), 1317-1441.
137. Journal of the American Medical Association. (1994). The 2nd International Congress on Peer Review in Biomedical Publication. Proceedings. Journal of the American Medical Association, 272(2), 79-174.
138. Journal of the American Medical Association. (1998). The International Congress on Biomedical Peer Review. Journal of the American Medical Association, 280(3), 203-306.
139. Journal of the American Medical Association. (2002). The International Congress on Biomedical Peer Review. Journal of the American Medical Association, 287(21), 2759-2871.
140. Koch, S. (2006). Die Begutachtungsverfahren der Deutschen Forschungsgemeinschaft nach Einführung der Fachkollegien. In S. Hornbostel & D. Simon (Eds.), Wie viel (In-) Transparenz ist notwendig Peer Review revisited (Vol. 1, pp. 15-26). Bonn, Germany: Institut für Forschungsinformation und Qualitätssicherung.
141. Kostoff, R. N. (1995). Federal research impact assessment: Axioms, approaches, applications. Scientometrics, 34(2), 163-206.
142. Kostoff, R. N. (1997). The principles and practices of peer review. Science and Engineering Ethics, 3(1), 19-34.
143. Kostoff, R. N. (2004). Research program peer review: Purposes, principles, practices, protocols (DTIC Technical Report Number ADA424141). Arlington, VA: Office of Naval Research.
144. Krampen, G., & Montada, L. (2002). Wissenschaftsforschung in der Psychologie. Göttingen, Germany: Hogrefe.
145. Kronick, D. A. (1990). Peer review in 18th century scientific journalism. Journal of the American Medical Association, 263(10), 1321-1322.
146. Kuhn, T. S. (1962). The structure of scientific revolutions. Chicago: University of Chicago Press.
147. Kupfersmid, J. (1988). Improving what is published: A model in search of an editor. American Psychologist, 43(8), 635-642.
148. LaFollette, M. C. (1992). Stealing into print: Fraud, plagiarism and misconduct in scientific publishing. Berkeley: University of California Press.
149. Lamont, M. (2009). How professors think: Inside the curious world of academic judgment. Cambridge, MA: Harvard University Press.
150. Ledin, A., Bornmann, L., Gannon, F., & Wallon, G. (2007). A persistent problem: Traditional gender roles hold back female scientists. EMBO Reports, 8(11), 982-987.
151. Lee, K. P., Boyd, E. A., Holroyd-Leduc, J. M., Bacchetti, P., & Bero, L. A. (2006). Predictors of publication: Characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Medical Journal of Australia, 184(12), 621-626.
152. Levenson, H., Burford, B., Bonno, B., & Davis, L. (1975). Are women still prejudiced against women A replication and extension of Goldberg's Study. Journal of Psychology, 89(1), 67-71.
153. Lienert, G. A. (1987). Schulnoten-Evaluation. Frankfurt am Main, Germany: Athenäum.
154. Lindsey, D. (1989). Using citation counts as a measure of quality in science: Measuring what's measurable rather than what's valid. Scientometrics, 15(3-4), 189-203.
155. Lloyd, M. E. (1990). Gender factors in reviewer recommendations for manuscript publication. Journal of Applied Behavioral Analysis, 23(4), 539-543.
156. Lock, S. (1985). A difficult balance: Editorial peer review in medicine. Philadelphia, PA: ISI Press.
157. Long, J. S., & Fox, M. F. (1995). Scientific careers: Universalism and particularism. Annual Review of Sociology, 21, 45-71.
158. Mahoney, M. J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 2, 161-175.
159. Mallard, G., Lamont, M., & Guetzkow, J. (2009). Fairness as appropriateness: Negotiating epistemological differences in peer review. Science, Technology, & Human Values, 34(5), 573-606.
160. Marsh, H., & Bornmann, L. (2009). Do women have less success in peer review Nature, 459, 602.
161. Marsh, H. W., & Ball, S. (1981). Interjudgmental reliability of reviews for the Journal of Educational Psychology. Journal of Educational Psychology, 73(6), 872-880.
162. Marsh, H. W., & Ball, S. (1991). Reflections on the peer review process. Behavioral and Brain Sciences, 14(1), 157-158.
163. Marsh, H. W., Bonds, N. W., & Jayasinghe, U. W. (2007). Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid. Australian Psychologist, 42(1), 33-38.
164. Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H.D., & O'Mara, A. (in press). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research.
165. Marsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008). Improving the peer-review process for grant applications: Reliability, validity, bias, and generalizability. American Psychologist, 63(3), 160-168.
166. Martin, B. (2000). Research grants: Problems and options. Australian Universities' Review, 43(2), 17-22.
167. Martin, B. R., & Irvine, J. (1983). Assessing basic research: Some partial indicators of scientific progress in radio astronomy. Research Policy, 12(2), 61-90.
168. Martinson, B. C., Anderson, M. S., Crain, A. L., & de Vries, R. (2006). Scientists' perceptions of organizational justice and self-reported misbehaviors. Journal of Empirical Research on Human Research Ethics, 1(1), 51-66.
169. Matt, G. E., & Navarro, A. M. (1997). What meta-analyses have and have not taught us about psychotherapy effects: A review and future directions. Clinical Psychology Review, 17(1), 1-32.
170. McClellan, J. E. (2003). Specialist control: The publications committee of the Academie Royal des Sciences (Paris) 1700-1793 (Transactions of the American Philosophical Society, v. 93). Philadelphia, PA: American Philosophical Society.
171. McCullough, J. (1989). First comprehensive survey of NSF applicants focuses on their concerns about proposal review. Science, Technology, & Human Values, 14(1), 78-88.
172. McCullough, J. (1994). The role and influence of the US National Science Foundation's program officers in reviewing and awarding grants. Higher Education, 28(1), 85-94.
173. McDonald, R. J., Cloft, H. J., & Kallmes, D. F. (2007). Fate of submitted manuscripts rejected from the American Journal of Neuroradiology: Outcomes and commentary. American Journal of Neuroradiology, 28(8), 1430-1434.
174. McDonald, R. J., Cloft, H. J., & Kallmes, D. F. (2009). Fate of manuscripts previously rejected by the American Journal of Neuroradiology: A follow-up analysis. American Journal of Neuroradiology, 30(2), 253-256.
175. McIntosh, E. G., & Ross, S. (1987). Peer review in psychology: Institutional ranking as a factor. Psychological Reports, 60(3), 1049-1050.
176. Melin, G., & Danell, R. (2006). The top eight percent: Development of approved and rejected applicants for a prestigious grant in Sweden. Science and Public Policy, 33(10), 702-712.
177. Merton, R. K. (1942). Science and technology in a democratic order. Journal of Legal and Political Sociology, 1, 115-126.
178. Merton, R. K. (1948). The self-fulfilling prophecy. Antioch Review, 8, 193-210.
179. Merton, R. K. (1968). The Matthew effect in science. Science, 159(3810), 56-63.
180. Moed, H. (2008). UK Research Assessment Exercises: Informed judgments on research quality or quantity Scientometrics, 74(1), 153-161.
181. National Academy of Sciences. (2006). Beyond bias and barriers: Fulfilling the potential of women in academic science and engineering. Washington, DC: The National Academies Press.
182. Neuhaus, C., & Daniel, H.D. (2009). A new reference standard for citation analysis in chemistry and related fields based on the sections of Chemical Abstracts. Scientometrics, 78(2), 219-229.
183. Nicolaisen, J. (2007). Citation analysis. Annual Review of Information Science and Technology, 41, 609-641.
184. Nylenna, M., Riis, P., & Karlsson, Y. (1994). Multiple blinded reviews of the same two manuscripts: Effects of referee characteristics and publication language. Journal of the American Medical Association, 272(2), 149-151.
185. Opthof, T., Furstner, F., van Geer, M., & Coronel, R. (2000). Regrets or no regrets No regrets! The fate of rejected manuscripts. Cardiovascular Research, 45(1), 255-258.
186. Opthof, T., & Wilde, A. A. M. (2009). The Hirsch-index: A simple, new tool for the assessment of scientific output of individual scientists: The case of Dutch professors in clinical cardiology. Netherlands Heart Journal, 17(4), 145-154.
187. Over, R. (1996). Perceptions of the Australian Research Council Large Grants Scheme: Differences between successful and unsuccessful applicants. Australian Educational Researcher, 23(2), 17-36.
188. Overbeke, J., & Wager, E. (2003). The state of the evidence: What we know and what we don't know about journal peer review. In F. Godlee & T. Jefferson (Eds.), Peer review in health sciences (2nd ed., pp. 45-61). London: BMJ Books.
189. Overview: Nature's peer review trial. (2006). Nature. Retrieved May 18, from www.nature.com/nature/peerreview/debate/nature05535.html
190. Owen, R. (1982). Reader bias. Journal of the American Medical Association, 247(18), 2533-2534.
191. Paludi, M. A., & Bauer, W. D. (1983). Goldberg revisited: What's in an author's name. Sex Roles, 9(3), 387-390.
192. Paludi, M. A., & Strayer, L. A. (1985). What's in an author's name Differential evaluations of performance as a function of author's name. Sex Roles, 12(3-4), 353-361.
193. Patterson, S. C., Bailey, M. S., Martinez, V. J., & Angel, S. C. (1987). Report of the managing editor of the American Political Science Review, 1986-1987. PS, 20, 1006-1016.
194. Pendlebury, D. A. (2008). Using bibliometrics in evaluating research. Philadelphia, PA: Research Department, Thomson Scientific.
195. Perneger, T. V. (2004). Relation between online "hit counts" and subsequent citations: Prospective study of research papers in the BMJ. British Medical Journal, 329, 546-547.
196. Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of accepted, published articles, submitted again. Behavioral and Brain Sciences, 5(2), 187-195.
197. Petty, R. E., & Fleming, M. A. (1999). The review process at PSPB: Correlates of interreviewer agreement and manuscript acceptance. Personality and Social Psychology Bulletin, 25(2), 188-203.
198. Pfeffer, J., Leong, A., & Strehl, K. (1977). Paradigm development and particularism: Journal publication in three scientific disciplines. Social Forces, 55(4), 938-951.
199. Pierie, J. P. E. N., Walvoort, H. C., & Overbeke, A. J. P. M. (1996). Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde. Lancet, 348(9040), 1480-1483.
200. Popper, K. R. (1961). The logic of scientific discovery (2nd ed.). New York: Basic Books.
201. Pöschl, U. (2004). Interactive journal concept for improved scientific publishing and quality assurance. Learned Publishing, 17(2), 105-113.
202. Pruthi, S., Jain, A., Wahid, A., Mehra, K., & Nabi, S. A. (1997). Scientific community and peer review system: A case study of a central government funding scheme in India. Journal of Scientific & Industrial Research, 56(7), 398-407.
203. Publishing Research Consortium. (2008). Peer review in scholarly journals: Perspective of the scholarly community: An international study. Bristol, UK: The Consortium.
204. Ray, J., Berkwits, M., & Davidoff, F. (2000). The fate of manuscripts rejected by a general medical journal. American Journal of Medicine, 109(2), 131-135.
205. Resnik, D. B., Gutierrez-Ford, C., & Peddada, S. (2008). Perceptions of ethical problems with scientific journal peer review: An exploratory study. Science and Engineering Ethics, 14(3), 305-310.
206. Ross, P. F. (1980). The sciences' self-management: Manuscript refereeing, peer review, and goals in science. Lincoln, MA: The Ross Company.
207. Rossiter, J. R. (2003). Qualifying the importance of findings. Journal of Business Research, 56(1), 85-88.
208. Roy, R. (1985). Funding science: The real defects of peer-review and an alternative to it. Science, Technology, & Human Values, 52, 73-81.
209. Russo, G. (2008). Statistical analyses raise questions over NIH grant reviews. Nature, 454, 801.
210. Sahner, H. (1982). On the selectivity of editors: An input-output analysis of the Zeitschrift für Soziologie. Zeitschrift für Soziologie, 11(1), 82-98.
211. Schuster, C. (2002). A mixture model approach to indexing rater agreement. British Journal of Mathematical & Statistical Psychology, 55, 289-303.
212. Scott, W. A. (1974). Interreferee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology. American Psychologist, 29(9), 698-702.
213. Seglen, P. O. (1994). Causal relationship between article citedness and journal impact. Journal of the American Society for Information Science, 45(1), 1-11.
214. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin Company.
215. Sharp, D. W. (1990). What can and should be done to reduce publication bias: The perspective of an editor. Journal of the American Medical Association, 263(10), 1390-1391.
216. Shashok, K. (2005). Standardization vs diversity: How can we push peer review research forward Medscape General Medicine, 7(1), 11.
217. Shatz, D. (2004). Peer review: A critical inquiry. Lanham, MD: Rowman & Littlefield.
218. Siegelman, S. S. (1991). Assassins and zealots: Variations in peer review: Special report. Radiology, 178(3), 637-642.
219. Sismondo, S. (1993). Some social constructions. Social Studies of Science, 23(3), 515-553.
220. Skrondal, A., & Rabe-Hesketh, S. (2004). Generalized latent variable modeling: Multilevel, longitudinal, and structural equation models. London: Chapman & Hall/CRC.
221. Smigel, E. O., & Ross, H. L. (1970). Factors in editorial decision. American Sociologist, 5(1), 19-21.
222. Smith, L. C. (1981). Citation analysis. Library Trends, 30(1), 83-106.
223. Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178-182.
224. Sokal, A. D. (2008). Beyond the hoax: Science, philosophy and culture. Oxford, UK: Oxford University Press.
225. Speck, B. W. (1993). Publication peer review: An annotated bibliography. Westport, CT: Greenwood Press.
226. Stamps, A. E. (1997). Advances in peer review research: An introduction. Science and Engineering Ethics, 3(1), 3-10.
227. Stehbens, W. E. (1999). Basic philosophy and concepts underlying scientific peer review. Medical Hypotheses, 52(1), 31-36.
228. Sternberg, R. J., Hojjat, M., Brigockas, M. G., & Grigorenko, E. L. (1997). Getting in: Criteria for acceptance of manuscripts in Psychological Bulletin, 1993-1996. Psychological Bulletin, 121(2), 321-323.
229. Stieg Dalton, M. F. (1995). Refereeing of scholarly works for primary publishing. Annual Review of Information Science and Technology, 30, 213-250.
230. Stricker, L. J. (1991). Disagreement among journal reviewers: No cause for undue alarm. Behavioral and Brain Sciences, 14(1), 163-164.
231. Suls, J., & Martin, R. (2009). The air we breathe: A critical look at practices and alternatives in the peer review process. Perspectives on Psychological Science, 4(1), 40-50.
232. Tight, M. (2003). Reviewing the reviewers. Quality in Higher Education, 9(3), 295-303.
233. Tregenza, T. (2002). Gender bias in the refereeing process Trends in Ecology & Evolution, 17(8), 349-350.
234. U.S. General Accounting Office (1999). Peer review practices at federal science agencies vary. Washington, DC: The Office.
235. U.S. Office of Management and Budget. (2004). Revised information quality bulletin for peer review. Washington, DC: The Office.
236. van den Besselaar, P., & Leydesdorff, L. (2007). Past performance as predictor of successful grant applications. A case study. Den Haag, The Netherlands: Rathenau Instituut.
237. van Raan, A. F. J. (1996). Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises. Scientometrics, 36(3), 397-420.
238. van Raan, A. F. J. (1999). Advanced bibliometric methods for the evaluation of universities. Scientometrics, 45(3), 417-423.
239. von Eye, A., & Mun, E. Y. (2005). Analyzing rater agreement: Manifest variable methods. Mahwah, NJ: Erlbaum.
240. Wakin, M., Rozell, C., Davenport, M., & Laska, J. (2009). Letter from the editors. Rejecta Mathematica, 1(1), 1-3.
241. Ward, C. (1981). Prejudice against women: Who, when, and why Sex Roles, 7(2), 163-171.
242. Wellcome Trust. (2001). Review of Wellcome Trust PhD research training: Career paths of a 1988 - 1990 prize student cohort. London: The Trust.
243. Weller, A. C. (1996). Editorial peer review: A comparison of authors publishing in two groups of US medical journals. Bulletin of the Medical Library Association, 84(3), 359-366.
244. Weller, A. C. (2002). Editorial peer review: Its strengths and weaknesses. Medford, NJ: Information Today, Inc.
245. Wennerås, C., & Wold, A. (1997). Nepotism and sexism in peer-review. Nature, 387(6631), 341-343.
246. Wessely, S. (1998). Peer review of grant applications: What do we know Lancet, 352(9124), 301-305.
247. Wessely, S., & Wood, F. (1999). Peer review of grant applications: A systematic review. In F. Godlee & T. Jefferson (Eds.), Peer Review in Health Sciences (pp. 14-31). London: BMJ Books.
248. White, H. D. (2005). On extending informetrics: An opinion paper. Proceedings of the 10th International Conference of the International Society for Scientometrics and Informetrics, 2, 442-449.
249. Whitley, R., & Gläser, J. (Eds.). (2007). The changing governance of the sciences: The advent of research evaluation systems. Dordrecht, the Netherlands: Springer.
250. Whitman, N., & Eyre, S. (1985). The pattern of publishing previously rejected articles in selected journals. Family Medicine, 17(1), 26-28.
251. Wiener, S., Urivetsky, M., Bregman, D., Cohen, J., Eich, R., Gootman, N., et al. (1977). Peer review: Inter-reviewer agreement during evaluation of research grant evaluations. Clinical Research, 25, 306-311.
252. Wiley, S. (2008). Peer review isn't perfect ... But it's not a conspiracy designed to maintain the status quo. The Scientist, 22(11), 31.
253. Wilson, J. D. (1978). Peer review and publication. Journal of Clinical Investigation, 61(4), 1697-1701.
254. Wood, F. Q., & Wessely, S. (2003). Peer review of grant applications: A systematic review. In F. Godlee & T. Jefferson (Eds.), Peer review in health sciences (2nd ed., pp. 14-44). London: BMJ Books.
255. Yalow, R. S. (1982). Is subterfuge consistent with good science Bulletin of Science Technology & Society, 2(5), 401-404.
256. Young, S. N. (2003). Peer review of manuscripts: Theory and practice. Journal of Psychiatry & Neuroscience, 28(5), 327-330.
257. Ziman, J. (2000). Real science. What it is, and what it means. Cambridge, UK: Cambridge University Press.
258. Zuckerman, H., & Merton, R. K. (1971a). Patterns of evaluation in science: Institutionalisation, structure and functions of the referee system. Minerva, 9(1), 66-100.
259. Zuckerman, H., & Merton, R. K. (1971b). Sociology of refereeing. Physics Today, 24(7), 28-33.