Inter-rater and test-retest reliability of quality assessments by novice student raters using the Jadad and Newcastle-Ottawa Scales

BMJ Open

Volumn 2, Issue 4, 2012, Pages

  Oremus, Mark ^a   Oremus, Carolina ^a,b   Hall, Geoffrey B C ^a,b   McKinnon, Margaret C ^a,b   Graham, Allyson ^a,b   Gregory, Caitlin ^a,b   Fervaha, Gagan ^a,b   Hanford, Lindsay ^a,b   Nazarov, Anthony ^a,b   Parlar, Melissa ^a,b   Restivo, Maria ^a,b   Tatham, Erica ^a,b   Truong, Wanda ^a,b  

^a MCMASTER UNIVERSITY   (Canada)

^b McMaster Integrative Neuroscience Discovery and Study ^*  (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; CASE CONTROL STUDY; CLINICAL ASSESSMENT TOOL; COGNITIVE DEFECT; COHORT ANALYSIS; CONTROLLED STUDY; CORRELATION COEFFICIENT; ELECTROCONVULSIVE THERAPY; HUMAN; INTER RATER RELIABILITY; JADAD SCALE; MAJOR DEPRESSION; NEWCASTLE OTTAWA SCALE; OBSERVATIONAL STUDY; QUALITY CONTROL; RANDOMIZED CONTROLLED TRIAL; RELIABILITY; SCORING SYSTEM; STUDENT; TEST RETEST RELIABILITY;

EID: 84865122567     PISSN: None     EISSN: 20446055     Source Type: Journal    
DOI: 10.1136/bmjopen-2012-001368     Document Type: Article

References (33)

1. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009;339:b2535.
2. Agency for Healthcare Research and Quality (AHRQ). Systems to Rate the Strength of Scientific Evidence. Evidence Report/Technology Assessment No. 47. Rockville, MD: Agency for Healthcare Research and Quality, 2002.
3. Verhagen AP, de Vet HC, de Bie RA, et al. The art of quality assessment of RCTs included in systematic reviews. J Clin Epidemiol 2001;54:651-4. (Pubitemid 32594850)
4. Oxman AD, Guyatt GH. Guidelines for reading literature reviews. CMAJ 1988;138:697-703.
5. Wells GA, Shea B, O'Connell D, et al. The Newcastle-Ottawa Scale (NOS) for Assessing the Quality of Nonrandomised Studies in Meta-analyses. Ottawa: Ottawa Hospital Research Institute. http://www.ohri.ca/programs/clinical- epidemiology/oxford.asp (accessed 2 Apr 2012).
6. Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996;17:1-12. (Pubitemid 26120810)
7. Chalmers TC, Smith H Jr, Blackburn B, et al . A method for assessing the quality of a randomized control trial. Control Clin Trials 1981;2:31-49. (Pubitemid 11083624)
8. Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0. [Updated March 2011]. The Cochrane Collaboration, 2011. http://www.cochrane-handbook.org (accessed 23 Apr 2012).
9. Maher CG, Sherrington C, Herbert RD, et al. Reliability of the PEDro scale for rating quality of randomized controlled trials. Phys Ther 2003;83:713-21. (Pubitemid 37238882)
10. Kocsis JH, Gerber AJ, Milrod B, et al. A new scale for assessing the quality of randomized clinical trials of psychotherapy. Compr Psychiatry 2010;51:319-24.
11. Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 1998;52:377-84. (Pubitemid 28438523)
12. Al-Harbi K, Farrokhyar F, Mulla S, et al. Classification and appraisal of the level of clinical evidence of publications from the Canadian Association of Pediatric Surgeons for the past 10 years. J Pediatr Surg 2009;44:1013-17.
13. Berard A, Andreu N, Tetrault J, et al. Reliability of Chalmers' scale to assess quality in meta-analyses on pharmacological treatments for osteoporosis. Ann Epidemiol 2000;10:498-503.
14. Hartling L, Ospina M, Liang Y, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ 2009;339:b4012.
15. Hartling L, Bond K, Vandermeer B, et al. Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma. PLoS One 2011;6:e17242.
16. Tooth L, Bennett S, McCluskey A, et al. Appraising the quality of randomized controlled trials: inter-rater reliability for the OTseeker evidence database. J Eval Clin Pract 2005;11:547-55. (Pubitemid 41789738)
17. Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001;135:982-9. (Pubitemid 33131619)
18. Hartling L, Hamm M, Milne A, et al. Validity and Inter-Rater Reliability Testing of Quality Assessment Instruments. Rockville, MD: Agency for Healthcare Research and Quality, 2012.
19. Oremus M, Wolfson C, Perrault A, et al. Interrater reliability of the modified Jadad quality scale for systematic reviews of Alzheimer's disease drug trials. Dement Geriatr Cogn Disord 2001;12:232-6. (Pubitemid 32217854)
20. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37-46.
21. Fleiss JL, Levin B, Paik MC. Statistical Methods for Rates and Proportions. 3rd edn. Hoboken, NJ: John Wiley & Sons, 2003.
22. Altman DG. Practical Statistics for Medical Research. London: Chapman and Hall, 1991.
23. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979;86:420-8.
24. Streiner DL, Norman GR. Health Measurement Scales: A Practical Guide to Their Development and Use. 4th edn. Oxford: Oxford University Press, 2008.
25. Fleiss J. The Design and Analysis of Clinical Experiments. New York: John Wiley & Sons, 1986.
26. Shamliyan T, Kane RL, Dickinson S. A systematic review of tools used to assess the quality of observational studies that examine incidence or prevalence and risk factors for diseases. J Clin Epidemiol 2010;63:1061-70.
27. Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 2007;36:666-76. (Pubitemid 47244573)
28. Lang S, Kleijnen J. Quality assessment tools for observational studies: lack of consensus. Int J Evid Based Healthc 2010;8:247.
29. Juurlink DN, Detsky AS. Kappa statistic. CMAJ 2005;173:16.
30. Agency for Healthcare Research and Quality (AHRQ). Effective Health Care Program. Rockville, MD: Agency for Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov (accessed 2 Apr 2012).
31. Horton J, Vandermeer B, Hartling L, et al. Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy. J Clin Epidemiol 2010;63:289-98.
32. Haywood KL, Hargreaves J, White R, et al. Reviewing measures of outcome: reliability of data extraction. J Eval Clin Pract 2004;10:329-37. (Pubitemid 38823236)
33. Higgins JPT, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928.