Tools used to assess the quality of peer review reports: A methodological systematic review

BMC Medical Research Methodology

Volumn 19, Issue 1, 2019, Pages

  Superchi, Cecilia ^a,b,c   González, José Antonio ^a   Solà, Ivan ^d,e   Cobo, Erik ^a   Hren, Darko ^f   Boutron, Isabelle ^g  

^a UNIVERSITAT POLITÈCNICA DE CATALUNYA   (Spain)

^b PARIS DESCARTES UNIVERSITY   (France)

^c UNIVERSITÉ PARIS DESCARTES   (France)

^d HOSPITAL DE LA SANTA CREU I SANT PAU   (Spain)

^e CIBER EPIDEMIOLOGÍA Y SALUD PÚBLICA CIBERESP   (Spain)

^f UNIVERSITY OF SPLIT   (Croatia)

^g HÔTEL DIEU   (France)

Author keywords

Methods; Peer review; Quality control; Report; Systematic review

Indexed keywords

CHECKLIST; EVIDENCE BASED MEDICINE; HUMAN; MEDICAL RESEARCH; METHODOLOGY; PEER REVIEW; PROCEDURES; RANDOMIZED CONTROLLED TRIAL (TOPIC); REPRODUCIBILITY; RESEARCH;

BIOMEDICAL RESEARCH; CHECKLIST; EVIDENCE-BASED MEDICINE; HUMANS; OUTCOME ASSESSMENT, HEALTH CARE; PEER REVIEW; RANDOMIZED CONTROLLED TRIALS AS TOPIC; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; RESEARCH REPORT;

EID: 85062586466     PISSN: None     EISSN: 14712288     Source Type: Journal    
DOI: 10.1186/s12874-019-0688-x     Document Type: Review

References (55)

1. Kronick DA. Peer review in 18th-century scientific journalism. JAMA. 1990;263(10):1321-2.
2. Jefferson T, Alderson P, Wager E, Davidoff F. Effects of editorial peer review. JAMA. 2002;287(21):2784-6.
3. Smith R. Peer review: a flawed process at the heart of science and journals. J R Soc Med. 2006;99:178-82.
4. Baxt WG, Waeckerle JF, Berlin JA, Callaham ML. Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance. Ann Emerg Med. 1998;32(3):310-7.
5. Kravitz RL, Franks P, Feldman MD, Gerrity M, Byrne C, William M. Editorial peer reviewers' recommendations at a general medical journal: are they reliable and do editors care? PLoS One. 2010;5(4):2-6.
6. Yaffe MB. Re-reviewing peer review. Sci Signal. 2009;2(85):1-3.
7. Stahel PF, Moore EE. Peer review for biomedical publications: we can improve the system. BMC Med. 2014;12(179):1-4.
8. Rennie D. Make peer review scientific. Nature. 2016;535:31-3.
9. Moher D. Custodians of high-quality science: are editors and peer reviewers good enough? https://www.youtube.com/watch?v=RV2tknDtyDs&t=454s. Accessed 16 Oct 2017.
10. Ghimire S, Kyung E, Kang W, Kim E. Assessment of adherence to the CONSORT statement for quality of reports on randomized controlled trial abstracts from four high-impact general medical journals. Trials. 2012;13:77.
11. Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results. JAMA. 2010;303(20):2058-64.
12. Hopewell S, Collins GS, Boutron I, Yu L-M, Cook J, Shanyinde M, et al. Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study. BMJ. 2014;349:g4145.
13. Lazarus C, Haneef R, Ravaud P, Boutron I. Classification and prevalence of spin in abstracts of non-randomized studies evaluating an intervention. BMC Med Res Methodol. 2015;15:85.
14. Jefferson T, Rudin M, Brodney Folse S, et al. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev. 2007;2:MR000016.
15. Bruce R, Chauvin A, Trinquart L, Ravaud P, Boutron I. Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis. BMC Med. 2016;14:85.
16. Moher D, Liberati A, Tetzlaff J, Altman DG, Group TP. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.
17. NHS. PROSPERO International prospective register of systematic reviews. https://www.crd.york.ac.uk/prospero/. Accessed 6 Nov 2017.
18. Sanderson S, Tatt ID, Higgins JPT. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Intern J Epidemiol. 2007;36:666-76.
19. R Core Team. R: a language and environment for statistical computing. http://www.r-project.org/. Accessed 4 Dec 2017.
20. Gentles SJ, Charles C, Nicholas DB, Ploeg J, McKibbon KA. Reviewing the research methods literature: principles and strategies illustrated by a systematic overview of sampling in qualitative research. Syst Rev. 2016;5:172.
21. Glaser B, Strauss A. The discovery of grounded theory. Chicago: Aldine; 1967.
22. Friedman DP. Manuscript peer review at the AJR: facts, figures, and quality assessment. Am J Roentgenol. 1995;164(4):1007-9.
23. Black N, Van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review for a general medical journal? JAMA. 1998;280(3):231-3.
24. Henly SJ, Dougherty MC. Quality of manuscript reviews in nursing research. Nurs Outlook. 2009;57(1):18-26.
25. Callaham ML, Baxt WG, Waeckerle JF, Wears RL. Reliability of editors' subjective quality ratings of peer reviews of manuscripts. JAMA. 1998;280(3):229-31.
26. Callaham ML, Knopp RK, Gallagher EJ. Effect of written feedback by editors on quality of reviews: two randomized trials. JAMA. 2002;287(21):2781-3.
27. Van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers ' recommendations: a randomised trial. BMJ. 1999;318(7175):23-7.
28. Mcnutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. JAMA. 1990;263(10):1371-6.
29. Moore A, Jones R. Supporting and enhancing peer review in the BJGP. Br J Gen Pract. 2014;64(624):e459-61.
30. Stossel TP. Reviewer status and review quality. N Engl J Med. 1985;312(10):658-9.
31. Thompson SR, Agel J, Losina E. The JBJS peer-review scoring scale: a valid, reliable instrument for measuring the quality of peer review reports. Learn Publ. 2016;29:23-5.
32. Rajesh A, Cloud G, Harisinghani MG. Improving the quality of manuscript reviews: impact of introducing a structured electronic template to submit reviews. AJR. 2013;200:20-3.
33. Shattell MM, Chinn P, Thomas SP, Cowling WR. Authors' and editors' perspectives on peer review quality in three scholarly nursing journals. J Nurs Scholarsh. 2010;42(1):58-65.
34. Jawaid SA, Jawaid M, Jafary MH. Characteristics of reviewers and quality of reviews: a retrospective study of reviewers at Pakistan journal of medical sciences. Pakistan J Med Sci. 2006;22(2):101-6.
35. Justice AC, Cho MK, Winker MA, Berlin JA. Does masking author identity improve peer review quality ? A randomized controlled trial. JAMA. 1998;280(3):240-3.
36. Henly SJ, Bennett JA, Dougherty MC. Scientific and statistical reviews of manuscripts submitted to nursing research: comparison of completeness, quality, and usefulness. Nurs Outlook. 2010;58(4):188-99.
37. Hettyey A, Griggio M, Mann M, Raveh S, Schaedelin FC, Thonhauser KE, et al. Peerage of science: will it work? Trends Ecol Evol. 2012;27(4):189-90.
38. Publons. Publons for editors: overview. https://static1.squarespace.com/static/576fcda2e4fcb5ab5152b4d8/t/58e21609d482e9ebf98163be/1491211787054/Publons-for-Editors-Overview.pdf. Accessed 20 Oct 2017.
39. Van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol. 1999;52(7):625-9.
40. Evans AT, McNutt RA, Fletcher SW, Fletcher RH. The characteristics of peer reviewers who produce good-quality reviews. J Gen Intern Med. 1993;8(8):422-8.
41. Feurer I, Becker G, Picus D, Ramirez E, Darcy M, Hicks M. Evaluating peer reviews: pilot testing of a grading instrument. JAMA. 1994;272(2):98-100.
42. Landkroon AP, Euser AM, Veeken H. Quality assessment of reviewers' reports using a simple instrument. Obstet Gynecol. 2006;108(4):979-85.
43. Greenland S, O'Rourke K. On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostatistics. 2001;2(4):463-71.
44. Jüni P, Witschi A, Bloch R. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282(11):1054-60.
45. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.
46. Schroter S, Price A, Flemyng E, et al. Perspectives on involvement in the peer-review process: surveys of patient and public reviewers at two journals. BMJ Open. 2018;8:e023357.
47. Ćurković M, Košec A. Bubble effect: including internet search engines in systematic reviews introduces selection bias and impedes scientific reproducibility. BMC Med Res Methodol. 2018;18(1):130.
48. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156-65.
49. Kliewer MA, Freed KS, DeLong DM, Pickhardt PJ, Provenzale JM. Reviewing the reviewers: comparison of review quality and reviewer characteristics at the American journal of roentgenology. AJR. 2005;184(6):1731-5.
50. Berquist T. Improving your reviewer score: it's not that difficult. AJR. 2017;209:711-2.
51. Callaham ML, Mcculloch C. Longitudinal trends in the performance of scientific peer reviewers. Ann Emerg Med. 2011;57(2):141-8.
52. Yang Y. Effects of training reviewers on quality of peer review: a before-and-after study (Abstract). https://peerreviewcongress.org/abstracts-2009.html. Accessed 7 Nov 2017.
53. Prechelt L. Review quality collector. https://reviewqualitycollector.org/static/pdf/rqdef-example.pdf. Accessed 20 Oct 2017.
54. Das Sinha S, Sahni P, Nundy S. Does exchanging comments of Indian and non-Indian reviewers improve the quality of manuscript reviews? Natl Med J India. 1999;12(5):210-3.
55. Callaham ML, Schriger DL. Effect of structured workshop training on subsequent performance of journal peer reviewers. Ann Emerg Med. 2002;40(3):323-8.