Quality assessment of observational studies in a drug-safety systematic review, Comparison of two tools: The Newcastle-Ottawa scale and the RTI item bank

Clinical Epidemiology

Volumn 6, Issue , 2014, Pages 981-993

  Margulis, Andrea V ^a   Pladevall, Manel ^a   Riera guardia, Nuria ^a   Varas lorenzo, Cristina ^a   Hazell, Lorna ^b,c   Berkman, Nancy D ^d   Viswanathan, Meera ^d   Perez gutthann, Susana ^a  

^a RTI HEALTH SOLUTIONS   (United States)

^b DRUG SAFETY RESEARCH UNIT   (United Kingdom)

^c UNIVERSITY OF PORTSMOUTH   (United Kingdom)

^d RTI INTERNATIONAL   (United States)

Author keywords

AC1; Meta analysis; Quality assessment; Systematic review

Indexed keywords

ANTIDIABETIC AGENT;

AGE; ARTICLE; DRUG SAFETY; FIRST ORDER AGREEMENT COEFFICIENT STATISTIC; HEALTH CARE ACCESS; HUMAN; INTERMETHOD COMPARISON; INTERRATER AGREEMENT; INTERRATER RELIABILITY; NAMED INVENTORIES, QUESTIONNAIRES AND RATING SCALES; NEWCASTLE OTTAWA SCALE; NON INSULIN DEPENDENT DIABETES MELLITUS; OBSERVATIONAL STUDY; PHARMACOEPIDEMIOLOGY; QUALITY CONTROL; RTI ITEM BANK; SEX; STATISTICAL ANALYSIS; SYSTEMATIC ERROR; SYSTEMATIC REVIEW (TOPIC);

EID: 84907862051     PISSN: None     EISSN: 11791349     Source Type: Journal    
DOI: 10.2147/CLEP.S66677     Document Type: Article

References (47)

1. Higgins JP., Green S, editors. Section 13.5.2.3. Tools for assessing methodological quality or risk of bias in non-randomized studies. In: Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. London: The Cochrane Collaboration; 2011.
2. Hartling L, Hamm M, Milne A, et al. Validity and Inter-Rater Reliability Testing of Quality Assessment Instruments (Report No: 12-EHC039-EF). Rockville, MD: Agency for Healthcare Research and Quality (US); 2012. Available from: http://www.ncbi.nlm.nih.gov/books/NBK92293/. Accessed June 24, 2013.
3. Greenland S, O’Rourke K. On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostatistics. 2001;2(4):463-471.
4. Neyarapally GA, Hammad TA, Pinheiro SP, Iyasu S. Review of quality assessment tools for the evaluation of pharmacoepidemiological safety studies. BMJ Open. 2012;2(5).
5. Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007;36(3):666-676.
6. Deeks JJ, Dinnes J, D’Amico R, et al; International Stroke Trial Collaborative Group; European Carotid Sugery Trial Collaborative Group. Evaluating non-randomised intervention studies. Health Technol Assess. 2003;7(27):iii-x, 1-173.
7. Wells GA, Shea B, O’Connell D, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses [webpage on the Internet]. Ottawa, ON: Ottawa Hospital Research Institute; 2011. Available from: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp. Accessed February 5, 2013.
8. SOS. Safety of NSAIDs [homepage on the Internet]. 2013. Available from: http://www.sos-nsaids-project.org/. Accessed June 24, 2013.
9. Castellsague J, Riera-Guardia N, Calingaert B, et al; Safety of Non-Steroidal Anti-Inflammatory Drugs (SOS) Project. Individual NSAIDs and upper gastrointestinal complications: a systematic review and meta-analysis of observational studies (the SOS project). Drug Saf. 2012;35(12):1127-1146.
10. Varas-Lorenzo C, Riera-Guardia N, Calingaert B, et al. Stroke risk and NSAIDs: a systematic review of observational studies. Pharmacoepidemiol Drug Saf. 2011;20(12):1225-1236.
11. Varas-Lorenzo C, Riera-Guardia N, Calingaert B, et al. Myocardial infarction and individual nonsteroidal anti-inflammatory drugs meta-analysis of observational studies. Pharmacoepidemiol Drug Saf. 2013;22(6):559-570.
12. SAFEGUARD. Safety evaluation of adverse reactions in diabetes [homepage on the Internet]. 2011. Available from: http://www.safeguard-diabetes.org/. Accessed June 24, 2013.
13. Viswanathan M, Berkman ND. Development of the RTI item bank on risk of bias and precision of observational studies [webpage on the Internet]. Rockville, MD: Agency for Healthcare Research and Quality; 2011. Available from: http://effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=784. Accessed February 5, 2013.
14. Viswanathan M, Berkman ND. Development of the RTI item bank on risk of bias and precision of observational studies. J Clin Epidemiol. 2012;65(2):163-178.
15. Varas-Lorenzo C, Riera-Guardia N, Calingaert B, et al. Cardiovascular risk with glitazones and metformin: results from a systematic review of observational studies. Pharmacoepidemiol Drug Saf. 2013;22(Suppl 1):S68.
16. Pladevall M, Varas-Lorenzo C, Margulis AV, et al. Rosiglitazone vs pioglitazone and acute myocardial infarction: systematic review and meta-analysis of observational studies. Pharmacoepidemiol Drug Saf. 2013;22(Suppl 1):S181-S182.
17. Stang A. Critical evaluation of the Newcastle-Ottawa Scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010;25(9):603-605.
18. Oremus M, Oremus C, Hall GB, McKinnon MC; ECT and Cognition Systematic Review Team. Inter-rater and test-retest reliability of quality assessments by novice student raters using the Jadad and Newcastle-Ottawa Scales. BMJ Open. 2012;2(4).
19. Hartling L, Milne A, Hamm MP, et al. Testing the Newcastle Ottawa Scale showed low reliability between individual reviewers. J Clin Epidemiol. 2013;66(9):982-993.
20. Al-Harbi K, Farrokhyar F, Mulla S, Fitzgerald P. Classification and appraisal of the level of clinical evidence of publications from the Canadian Association of Pediatric Surgeons for the past 10 years. J Pediatr Surg. 2009;44(5):1013-1017.
21. Tricco AC, Lillie E, Soobiah C, Perrier L, Straus SE. Impact of H1N1 on socially disadvantaged populations: systematic review. PLoS One. 2012;7(6):e39437.
22. Sunkara SK, Khairy M, El-Toukhy T, Khalaf Y, Coomarasamy A. The effect of intramural fibroids without uterine cavity involvement on the outcome of IVF treatment: a systematic review and meta-analysis. Hum Reprod. 2010;25(2):418-429.
23. Huang Y, Li YL, Huang H, Wang L, Yuan WM, Li J. Effects of hyperuricemia on renal function of renal transplant recipients: a systematic review and meta-analysis of cohort studies. PLoS One. 2012;7(6):e39457.
24. Uçeyler N, Häuser W, Sommer C. Systematic review with meta-analysis: cytokines in fibromyalgia syndrome. BMC Musculoskelet Disord. 2011;12:245.
25. Higgins JP, Altman DG, Gøtzsche PC, et al; Cochrane Bias Methods Group; Cochrane Statistical Methods Group. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.
26. Viswanathan M, Berkman ND, Dryden DM, Hartling L. Assessing risk of bias and confounding in observational studies of interventions or exposures: further development of the RTI item bank [webpage on the Internet]. Rockville, MD: Agency for Healthcare Research and Quality (US); 2013. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24006553. Accessed September 6, 2013.
27. Al-Saleh MA, Armijo-Olivo S, Thie N, et al. Morphologic and functional changes in the temporomandibular joint and stomatognathic system after transmandibular surgery in oral and oropharyngeal cancers: systematic review. J Otolaryngol Head Neck Surg. 2012;41(5):345-360.
28. Ruhe A, Fejer R, Walker B. Does postural sway change in association with manual therapeutic interventions? A review of the literature. Chiropr Man Therap. 2013;21(1):9.
29. Suissa S. Immortal time bias in observational studies of drug effects. Pharmacoepidemiol Drug Saf. 2007;16(3):241-249.
30. Walker AM. Confounding by indication. Epidemiology. 1996;7(4): 335-336.
31. Gwet KL. Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol. 2008;61(Pt 1): 29-48.
32. Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360-363.
33. Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013;13:61.
34. Brasil PE. Gwet’s AC1 interrater reliability [webpage on the Internet]. R-sig-Epi; 2012. Available from: https://stat.ethz.ch/pipermail/r-sig-epi/2012-May/000273.html. Accessed June 12, 2014.
35. Blood E, Spratt KF. Disagreement on Agreement: Two Alternative Agreement Coefficients. SAS Global Forum 2007: Statistics and Data Analysis. Paper 186-2007. Available from: http://www2.sas.com/proceedings/forum2007/186-2007.pdf. Accessed October 15, 2012.
36. da Costa BR, Hilfiker R, Egger M. PEDro’s bias: summary quality scores should not be used in meta-analysis. J Clin Epidemiol. 2013; 66(1):75-77.
37. Guyatt GH, Oxman AD, Vist G, et al. GRADE guidelines: 4. Rating the quality of evidence - study limitations (risk of bias). J Clin Epidemiol. 2011;64(4):407-415.
38. Mustafa RA, Santesso N, Brozek J, et al. The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses. J Clin Epidemiol. 2013;66(7):736-742; quiz 742. e1.
39. Lexchin J, Bero L., Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003;326(7400):1167-1170.
40. Sismondo S. Pharmaceutical company funding and its consequences: a qualitative systematic review. Contemp Clin Trials. 2008;29(2): 109-113.
41. Golder S, Loke YK. Is there evidence for biased reporting of published adverse effects data in pharmaceutical industry-funded studies? Br J Clin Pharmacol. 2008;66(6):767-773.
42. Schott G, Pachl H, Limbach U, Gundert-Remy U, Ludwig WD, Lieb K. The financing of drug trials by pharmaceutical companies and its consequences. Part 1: a qualitative, systematic review of the literature on possible influences on the findings, protocols, and quality of drug trials. Dtsch Arztebl Int. 2010;107(16):279-285.
43. Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L. Industry sponsorship and research outcome. Cochrane Database Syst Rev. 2012;12: MR000033.
44. Bero LA. Why the Cochrane risk of bias tool should include funding source as a standard item. Cochrane Database Syst Rev. 2013;12: ED000075.
45. Sterne JA. Why the Cochrane risk of bias tool should not include funding source as a standard item. Cochrane Database Syst Rev. 2013;12: ED000076.
46. Hernán MA. With great data comes great responsibility: publishing comparative effectiveness research in epidemiology. Epidemiology. 2011;22(3):290-291.
47. Margulis AV, Pladevall M, Riera-Guardia N, Varas-Lorenzo C, Hazell L, Perez-Gutthann S. Quality assessment of observational studies in a drug- safety systematic review, comparison of two tools: the Newcastle-Ottawa Scale and the RTI Item Bank. Presented at the 29th Annual ICPE meeting, Montreal, Canada. 2013. Pharmacoepidemiol Drug Saf. 2012;22(Suppl 1):S226.