In public peer review of submitted manuscripts, how do reviewer comments differ from comments written by interested members of the scientific community? A content analysis of comments written for Atmospheric Chemistry and Physics

Scientometrics

Volumn 93, Issue 3, 2012, Pages 915-929

  Bornmann, Lutz ^a   Herich, Hanna ^b,c   Joos, Hanna ^b   Daniel, Hans Dieter ^d,e  

^a MAX PLANCK INSTITUTE   (Germany)

^b Land Climate Dynamics Group   (Switzerland)

^c SWISS FEDERAL LABORATORIES FOR MATERIALS SCIENCE AND TECHNOLOGY   (Switzerland)

^d ETH ZURICH   (Switzerland)

^e UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

Content analysis; Criteria and reasons in manuscript review; Public peer review; Thematic areas for manuscript review

Indexed keywords

EID: 84869154860     PISSN: 01389130     EISSN: None     Source Type: Journal    
DOI: 10.1007/s11192-012-0731-8     Document Type: Article

References (41)

1. American Association for the Advancement of Science. (1998). Report in AAAS/UNESCO/ICSU Workshop on Developing Practices and Standards for Electronic Publishing in Science. Paris, October 12-14, 1998. Washington, DC: The American Association for the Advancement of Science.
2. Anon. (2006). Peer review on trial. Nature, 441(7094), 668.
3. Bakanic, V., McPhail, C., & Simon, R. J. (1989). Mixed messages: Referees' comments on the manuscripts they review. Sociological Quarterly, 30(4), 639-654.
4. Bedeian, A. G. (2004). Peer review and the social construction of knowledge in the management discipline. Academy of Management Learning and Education, 3(2), 198-216.
5. Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199-245.
6. Bornmann, L., & Daniel, H.-D. (2008). The effectiveness of the peer review process: Inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition, 47(38), 7173-7178. doi: 10. 1002/anie. 200800513.
7. Bornmann, L., Marx, W., Schier, H., Thor, A., & Daniel, H.-D. (2010a). From black box to white box at open access journals: Predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics. Research Evaluation, 19(2), 105-118.
8. Bornmann, L., Mutz, R., & Daniel, H.-D. (2008a). Latent Markov modeling applied to grant peer review. Journal of Informetrics, 2(3), 217-228.
9. Bornmann, L., Mutz, R., & Daniel, H.-D. (2009). The influence of the applicants' gender on the modeling of a peer review process by using latent Markov models. Scientometrics, 81(2), 407-411.
10. Bornmann, L., Nast, I., & Daniel, H.-D. (2008b). Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics, 77(3), 415-432.
11. Bornmann, L., Weymuth, C., & Daniel, H.-D. (2010b). A content analysis of referees' comments: How do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ? Scientometrics, 83(2), 493-506.
12. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
13. Daniel, H.-D. (1993). Guardians of science. Fairness and reliability of peer review. Weinheim: Wiley-VCH.
14. Fleiss, J. (1981). Statistical methods for rates and proportions. New York, NY: Wiley-VCH.
15. Gosden, H. (2003). 'Why not give us the full story?': Functions of referees' comments in peer reviews of scientific research papers. Journal of English for Academic Purposes, 2(2), 87-101.
16. Harnad, S. (1978). Inaugural editorial. Behavioral and Brain Sciences, 1(1).
17. Harnad, S. (2000). The invisible hand of peer review. Exploit Interactive (5).
18. Hosmer, D. W., & Lemeshow, S. (2000). Applied logistic regression (2nd ed.). Chichester: Wiley.
19. House of Commons; Science and Technology Committee on alternative models for the federal funding of science. (2011). Peer review in scientific publications. Eighth Report of Session 2010-12. Volume I: Report, together with formal minutes, oral and written evidence. London: House of Commons.
20. Janeway, C. A. J. (1990). JMCI: The last issue. Journal of Molecular and Cellular Immunology, 4, 293.
21. Krampen, G., Becker, R., Wahner, U., & Montada, L. (2007). On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications. Scientometrics, 71(2), 191-202.
22. Kumar, P., Rafiq, I., & Imam, B. (2011). Negotiation on the assessment of research articles with academic reviewers: Application of peer-review approach of teaching. Higher Education, 62(3), 315-332. doi: 10. 1007/s10734-010-9390-y.
23. Kupfersmid, J. (1988). Improving what is published: A model in search of an editor. American Psychologist, 43(8), 635-642.
24. Landis, J. R., & Koch, G. G. (1977). Measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174.
25. McCormack, N. (2009). Peer review and legal publishing: What law librarians need to know about open, single-blind, and double-blind reviewing. Law Library Journal, 101(1), 59-70.
26. Mizzaro, S. (2003). Quality control in scholarly publishing: Anew proposal. Journal of the American Society for Information Science and Technology, 54(11), 989-1005. doi: 10. 1002/Asi. 10296.
27. Morrison, J. (2006). The case for open peer review. Medical Education, 40(9), 830-831. doi: 10. 1111/j. 1365-2929. 2006. 02573. x.
28. Pöschl, U. (2004). Interactive journal concept for improved scientific publishing and quality assurance. Learned Publishing, 17(2), 105-113. doi: 10. 1087/095315104322958481.
29. Pöschl, U. (2010). Interactive open access publishing and peer review: The effectiveness and perspectives of transparency and self-regulation in scientific communication and evaluation. Liber Quarterly, 19(3/4), 293-314.
30. Rabe-Hesketh, S., & Everitt, B. (2004). A handbook of statistical analyses using Stata. Boca Raton, FL: Chapman & Hall/CRC.
31. Schultz, D. M. (2010). Rejection rates for journals publishing atmospheric science. Bulletin of the American Meteorological Society, 91(2), 231-243. doi: 10. 1175/2009BAMS2908. 1.
32. Shashok, K. (2008). Content and communication: How can peer review provide helpful feedback about the writing? BMC Medical Research Methods, 8(3).
33. Shum, S. B., & Sumner, T. (2001). JIME: An interactive journal for interactive media. First Monday, 6(2-5).
34. Smith, R. (1999). Opening up BMJ peer review. British Medical Journal, 318(7175), 4-5. doi: 10. 1136/bmj. 318. 7175. 4.
35. StataCorp. (2011). Stata statistical software: Release 12. College Station, TX: Stata Corporation.
36. Sumner, T., & Shum, S. B. (1996). Open peer review & argumentation: Loosening the paper chains on journals. ARIADNE, 5.
37. van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers' recommendations: Arandomised trial. British Medical Journal, 318(7175), 23-27.
38. von Eye, A., & Mun, E. Y. (2005). Analyzing rater agreement. Manifest variable methods. Mahwah, NJ: Lawrence Erlbaum Associates.
39. Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: Arandomised controlled trial. British Journal of Psychiatry, 176, 47-51.
40. Wing, J. M., & Chi, E. H. (2011). Reviewing peer review. Communications of the ACM, 54(7), 10-11. doi: 10. 1145/1965724. 1965728.
41. Xin, H. (2006). Online sleuths challenge cell paper. Science, 314(5806), 1669. doi: 10. 1126/science. 314. 5806. 1669a.