Reliability of Reviewer Ratings in the Manuscript Peer Review Process: An Opportunity for Improvement

Accountability in Research

Volumn 20, Issue 4, 2013, Pages 270-284

  Onitilo, Adedayo A ^a,b   Engel, Jessica M ^a   Salzman Scott, Sherry A ^a   Stankowski, Rachel V ^a   Doi, Suhail A R ^b  

^a MARSHFIELD CLINIC   (United States)

^b UNIVERSITY OF QUEENSLAND   (Australia)

Author keywords

accountability; disagreement; peer review; rating

Indexed keywords

ARTICLE; HUMAN; PEER REVIEW; REPRODUCIBILITY; SOCIAL BEHAVIOR; STANDARD; TOTAL QUALITY MANAGEMENT;

HUMANS; PEER REVIEW, RESEARCH; QUALITY IMPROVEMENT; REPRODUCIBILITY OF RESULTS; SOCIAL RESPONSIBILITY;

EID: 84880015710     PISSN: 08989621     EISSN: 15455815     Source Type: Journal    
DOI: 10.1080/08989621.2013.804345     Document Type: Article

References (29)

1. Abbott, A. 2011. A how-to for peer review. Nature, 473: 17
2. Armstrong, J. S. 1996. We need to rethink the editorial role of peer reviewers. Chronicle of Higher Education, 43: B3 - B4.
3. Blomme, E. A. 2011. The ARRIVE guidelines: A resource for authors and reviewers to ensure that submissions to The Veterinary Journal meet minimal expectations of completeness, accuracy and transparency. The Veterinary Journal, 189: 237 - 238.
4. Callaham, M. L. 2002. Research into peer review and scientific publication: Journals look in the mirror. Ann. Emerg. Med., 40: 313 - 316.
5. Campion, M. A. 1993. Article review checklist: A criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46: 705 - 718.
6. Curzon, M. E. and Cleaton-Jones, P. E. 2011. Reviewing scientific manuscripts. Eur. Arch. Paediatr. Dent., 12: 184 - 187.
7. Doi, S. A. (2011). Editors must have a mandate that excludes the tradition of confirmatory bias. October 18, 2011. http://www.bmj.com/rapid-response/2011/11/03/editors-must-have-mandate-excludes-tradition-confirmatory-bias (http://www.bmj.com/rapid-response/2011/11/03/editors-must-have-mandate-excludes-tradition-confirmatory-bias) (Accessed: 23 January 2013).
8. Duchesne, S. and Jannin, P. 2008. Proposing a manuscript peer-review checklist. Neuroimage, 39: 1783 - 1787.
9. Fiske, D. W. and Fogg, L. 1990. But the reviewers are making different criticisms of my paper!. American Psychologist, 45: 591 - 598.
10. Heddle, N. M. and Ness, P. M. 2009. Reviewing manuscripts: tips and responsibilities. Transfusion, 49: 2265 - 2268.
11. Henly, S. J. and Dougherty, M. C. 2009. Quality of manuscript reviews in nursing research. Nurs. Outlook, 57: 18 - 26.
12. Howard, L. and Wilkinson, G. 1998. Peer review and editorial decision-making. Br. J. Psychiatry, 173: 110 - 113. discussion 114-115
13. Ioannidis, J. P., Tatsioni, A. and Karassa, F. B. 2010. Who is afraid of reviewers' comments? Or, why anything can be published and anything can be cited. Eur. J. Clin. Invest., 40: 285 - 287.
14. Jackson, J. L., Srinivasan, M., Rea, J., Fletcher, K. E. and Kravitz, R. L. 2011. The validity of peer review in a general medicine journal. PLoS One, 6: e22475
15. Jefferson, T., Rudin, M., Brodney Folse, S. and Davidoff, F. 2007. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst. Rev, 2: MR000016
16. Landis, J. R. and Koch, G. G. 1977. The measurement of observer agreement for categorical data. Biometrics, 33: 159 - 174.
17. Loonen, M. P., Hage, J. J. and Kon, M. 2005. Who benefits from peer review? An analysis of the outcome of 100 requests for review by Plastic and Reconstructive Surgery. Plast. Reconstr. Surg., 116: 1461 - 1472. discussion 1473-1475
18. Mahoney, M. 1977. Publication prejudices: An experimental study of confirmatory bias in peer review system. Cognitive Therapy and Research, 1: 161 - 175.
19. Marotti, M. 2011. Primum non nocere: Authors or reviewers? Rev. Assoc. Med. Bras., 57 (477): 487
20. Neufeld, J. 1970. To amend refereeing. Physics Today, 23: 9 - 10.
21. Newton, D. P. 2010. Quality and peer review of research: an adjudicating role for. Account. Res., 17: 130 - 145.
22. Ploegh, H. 2011. End the wasteful tyranny of reviewer experiments. Nature, 472: 391
23. Prieto, L., Lamarca, R., Casado, A. and Alonso, J. 1997. The evaluation of agreement on continuous variables by the intraclass correlation coefficient. J. Epidemiol. Community Health, 51: 579 - 581.
24. Ramulu, V. G., Levine, R. B., Hebert, R. S. and Wright, S. M. 2005. Development of a case report review instrument. Int. J. Clin. Pract., 59: 457 - 461.
25. Rothwell, P. M. and Martyn, C. N. 2000. Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone?. Brain, 123: 1964 - 1969.
26. Schroter, S., Black, N., Evans, S., Carpenter, J., Godlee, F. and Smith, R. 2004. Effects of training on quality of peer review: Randomised controlled trial. BMJ, 328: 673
27. Sosa, J. A., Mehta, P., Thomas, D. C., Berland, G., Gross, C., McNamara, R. L., Rosenthal, R., Udelsman, R., Bravata, D. M. and Roman, S. A. 2009. Evaluating the surgery literature: can standardizing peer-review today predict manuscript impact tomorrow?. Ann. Surg., 250: 152 - 158.
28. Starbuck, W. H. 2003. Turning lemons into lemonade: Where is the value in peer reviews?. Journal of Management Inquiry, 12: 344 - 351.
29. Strayhorn, J. Jr, McDermott, J. F. Jr. and Tanguay, P. 1993. An intervention to improve the reliability of manuscript reviews for the. Journal of the American Academy of Child and Adolescent Psychiatry. Am. J. Psychiatry, 150: 947 - 952.