Inconsistency in the items included in tools used in general health research and physical therapy to evaluate the methodological quality of randomized controlled trials: A descriptive analysis

BMC Medical Research Methodology

Volumn 13, Issue 1, 2013, Pages

  Armijo Olivo, Susan ^a   Fuentes, Jorge ^a,b   Ospina, Maria ^a   Saltaji, Humam ^a   Hartling, Lisa ^a  

^a UNIVERSITY OF ALBERTA   (Canada)

^b UNIVERSIDAD CATÓLICA DEL MAULE   (Chile)

Author keywords

Bias; Critical appraisal; Methodological quality; Quality assessment; Quality of reporting; Risk of bias

Indexed keywords

HEALTH; HEALTH CARE QUALITY; HUMAN; METHODOLOGY; PHYSIOTHERAPY; RANDOMIZED CONTROLLED TRIAL (TOPIC); REVIEW; RISK; STATISTICAL BIAS; STATISTICAL MODEL; PROCEDURES;

BIAS (EPIDEMIOLOGY); HEALTH; HUMANS; MODELS, STATISTICAL; PHYSICAL THERAPY MODALITIES; QUALITY ASSURANCE, HEALTH CARE; RANDOMIZED CONTROLLED TRIALS AS TOPIC; RESEARCH DESIGN; RISK;

EID: 84883853641     PISSN: None     EISSN: 14712288     Source Type: Journal    
DOI: 10.1186/1471-2288-13-116     Document Type: Article

References (65)

1. Satge II. Conducting the Review. Phase 5 Study Quality Assessment. Khan K, Ter Riet G, Popay J, Nixon J, Kleijnen J, Undertaking Systematic Reviews of research Effectiveness CRD's Guidance for those carrying out or commissioning reviews York: Centre for Reviews and Dissemination 2001 1 20 20923587
2. The art of quality assessment of RCTs included in systematic reviews. Verhagen AP, de Vet HC, de Bie RA, Boers M, van den Brandt PA, J Clin Epidemiol 2001 54 7 651 654 10.1016/S0895-4356(00)00360-7 11438404 (Pubitemid 32594850)
3. Systematic reviews in health care: Assessing the quality of controlled clinical trials. Juni P, Altman DG, Egger M, BMJ 2001 323 7303 42 46 10.1136/bmj.323.7303.42 11440947 (Pubitemid 32682008)
4. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. Higgins JPT, Altman DG, Goetzsche PC, Juni P, Moher D, Oxman AD, Savovicć J, Schulz KF, Weeks L, Sterne JAC, BMJ 2011 343 7829 5928 10.1136/bmj.d5928 22008217
5. Scales to assess the quality of randomized controlled trials: a systematic review. Armijo-Olivo S, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ, Phys Ther 2008 88 2 156 175 10.2522/ptj.20070147 18073267 (Pubitemid 351210558)
6. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S, Control Clin Trials 1995 16 1 62 73 10.1016/0197-2456(94) 00031-W 7743790
7. Impact of quality scales on levels of evidence inferred from a systematic review of exercise therapy and low back pain. Colle F, Rannou F, Revel M, Fermanian J, Poiraudeau S, Arch Phys Med Rehabil 2002 83 12 1745 1752 10.1053/apmr.2002.35657 12474181 (Pubitemid 35424698)
8. Adjustment of meta-analyses on the basis of quality scores should be abandoned. Herbison P, Hay-Smith J, Gillespie WJ, J Clin Epidemiol 2006 59 12 1249 1256 17098567
9. Assessment of study quality for systematic reviews: a comparison of the Cochrane Collaboration Risk of Bias tool and the Effectve Public Health Practice Project Quality Assessment Tool: methodological research. Armijo-Olivo S, Stiles C, Hagen N, Biondo P, Cummings G, J Eval Clin Pract 2012 18 1 12 18 10.1111/j.1365-2753.2010.01516.x 20698919
10. Chapter 8: Assessing risk of bias in included studies. Higgins J, Altman D, Cochrane Handbook for Systematic Reviews of Interventions version 50 Chichester, UK: John Wiley & Sons, Ltd, Higgins J, Green S, 2008
11. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC, Int J Epidemiol 2007 36 4 847 857 10.1093/ije/dym087 17517809 (Pubitemid 47394153)
12. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: Meta-epidemiological study. Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJG, Sterne JAC, BMJ 2008 336 7644 601 605 10.1136/bmj.39465.451748.AD 18316340 (Pubitemid 351439518)
13. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Seida JK, Klassen TP, BMJ 2009 339 7728 1017
14. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP, Lancet 1998 352 9128 609 613 10.1016/S0140-6736(98)01085-X 9746022 (Pubitemid 28380142)
15. Empirical evidence of bias: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. Schulz KF, Chalmers I, Hayes RJ, Altman DG, JAMA 1995 273 5 408 412 10.1001/jama.1995. 03520290060030 7823387
16. Assessing risk of bias in included studies http://bmg.cochrane.org/ assessing-risk-bias-included-studies
17. Bias. Delgado-Rodriguez M, Llorca J, J Epidemiol Commun Health 2004 58 8 635 641 10.1136/jech.2003.008466
18. A systematic review of the content of critical appraisal tools. Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA, BMC Med Res Methodol 2004 4 1 22 10.1186/1471-2288-4-22 15369598 (Pubitemid 39532834)
19. Reviews assessing the quality or the reporting of randomized controlled trials are increasing over time but raised questions about how quality is assessed. Dechartres A, Charles P, Hopewell S, Ravaud P, Altman DG, J Clin Epidemiol 2011 64 2 136 144 10.1016/j.jclinepi.2010.04.015 20705426
20. A systematic review finds that methodological quality is better than its reputation but can be improved in physiotherapy trials in childhood cerebral palsy. Kunz R, Autti-Ramo I, Anttila H, Malmivaara A, Makela M, J Clin Epidemiol 2006 59 12 1239 1248 17098566
21. Validity. Streiner D, Norman G, Health Measurements Scales Oxford: Oxford University Press, Streiner D, Norman G, 2004 172 193
22. Reliability. Streiner D, Norman G, Health Measurements Scales Oxford: Oxford University Press, Streiner D, Norman G, 2004 126 152
23. Measuring Change. Streiner D, Norman G, Health Measurements Scales Oxford: Oxford University Press, Streiner D, Norman G, 2004 194 212
24. Quality criteria were proposed for measurement properties of health status questionnaires. Terwee CB, Bot SDM, de Boer MR, van der Windt DAWM, Knol DL, Dekker J, Bouter LM, de Vet HCW, J Clin Epidemiol 2007 60 1 34 42 10.1016/j.jclinepi.2006.03.012 17161752 (Pubitemid 44856236)
25. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Whiting P, Rutjes AWS, Reitsma JB, Glas AS, Bossuyt PMM, Kleijnen J, An Intern Med 2004 140 3 189 202 10.7326/0003-4819-140-3-200402030- 00010 (Pubitemid 38155650)
26. Bias in analytic research. Sackett DL, J Chronic Dis 1979 32 1-2 51 68 447779 (Pubitemid 10233847)
27. Development of the RTI item bank on risk of bias and precision of observational studies. Viswanathan M, Berkman ND, J Clin Epidemiol 2011 65 2 163 178 21959223
28. Evaluation of the quality of prognosis studies in systematic reviews. Hayden JA, Cotte P, Bombardier C, An Intern Med 2006 144 6 427 437 10.7326/0003-4819-144-6-200603210-00010 (Pubitemid 46768164)
29. Lash TL, Fox MP, Fink AK, Multiple Bias Modeling: Applying Quantitatvie Bias Analysis to Springer New York: Observational Epidemiologic Research 2009
30. How good is that agreement? Byrt T, Epidemiology (Cambridge, Mass) 1996 7 5 561
31. Development and evaluation of an instrument for the critical appraisal of randomized controlled trials of natural products. Jurgens T, Whelan AM, MacDonald M, Lord L, BMC Complem Altern Med 2009 9 11 doi:10.1186/1472-6882-9-11 10.1186/1472-6882-9-11
32. A quality-based review of randomized controlled trials of psychodynamic psychotherapy. Gerber AJ, Kocsis JH, Milrod BL, Roose SP, Barber JP, Thase ME, Perkins P, Leon AC, Am J Psychiatr 2011 168 1 19 28 10.1176/appi.ajp.2010. 08060843 20843868
33. A new scale for assessing the quality of randomized clinical trials of psychotherapy. Kocsis JH, Gerber AJ, Milrod B, Roose SP, Barber J, Thase ME, Perkins P, Leon AC, Compr Psychiatr 2010 51 3 319 324 10.1016/j.comppsych.2009. 07.001
34. Relationship between quality of reports of antidepressant randomized controlled trials and treatment estimates: Systematic review, meta-analysis, and meta-regression analysis. Cipriani A, Malvini L, Furukawa TA, Barbui C, J Clin Psychopharmacol 2007 27 4 352 356 10.1097/jcp.0b013e3180a9081f 17632218 (Pubitemid 47075505)
35. Development of a quality assessment instrument for trials of treatments for depression and neurosis. Moncrieff J, Churchill R, Colin Drummond D, McGuire H, Int J Meth Psychiatr Res 2001 10 3 126 133 10.1002/mpr.108 (Pubitemid 33014580)
36. A checklist to evaluate a report of a nonpharmacological trial (CLEAR NPT) was developed using consensus. Boutron I, Ravaud P, Moher D, Tugwell P, Giraudeau B, Poiraudeau S, Nizard R, J Clin Epidemiol 2005 58 12 1233 1240 10.1016/j.jclinepi.2005.05.004 16291467 (Pubitemid 41608539)
37. Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, McQuay HJ, Control Clin Trials 1996 17 1 1 12 10.1016/0197-2456(95) 00134-4 8721797 (Pubitemid 26120810)
38. Systematic reviews on the basis of methodological criteria. De Vet HCW, De Bie RA, Van Der Heijden GJMG, Verhagen AP, Sijpkes P, Knipschild PG, Physiotherapy 1997 83 6 284 289 10.1016/S0031-9406(05)66175-5 (Pubitemid 27279794)
39. The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. Verhagen AP, de Vet HC, de Bie RA, Kessels AG, Boers M, Bouter LM, Knipschild PG, J Clin Epidemiol 1998 51 12 1235 1241 10.1016/S0895-4356(98)00131-0 10086815 (Pubitemid 29009523)
40. PEDro: A database of randomized trials and systematic reviews in physiotherapy. Sherrington C, Herbert RD, Maher CG, Moseley AM, Man Ther 2000 5 4 223 226 10.1054/math.2000.0372 11052901
41. Evidence for physiotherapy practice: a survey of the Physiotherapy Evidence Database (PEDro). Moseley AM, Herbert RD, Sherrington C, Maher CG, Aust J Physiother 2002 48 1 43 49 11869164 (Pubitemid 34240086)
42. Method guidelines for systematic reviews in the Cochrane Collaboration Back Review Group for spinal disorders. Van Tulder MW, Assendelft WJJ, Koes BW, Bouter LM, Bombardier C, Nachemson AL, Esmail R, Deyo RA, Shekelle PG, Bouter LM, et al. Spine 1997 22 20 2323 2330 10.1097/00007632-199710150-00001 9355211 (Pubitemid 28069099)
43. Updated method guidelines for systematic reviews in the Cochrane Collaboration Back Review Group. Van Tulder M, Furlan A, Bombardier C, Bouter L, Spine 2003 28 12 1290 1299 12811274 (Pubitemid 36734620)
44. Systematic review of the quality of randomized controlled trials for patellofemoral pain syndrome. Bizzini M, Childs JD, Piva SR, J Orthop Sports Phys Ther 2003 33 1 4 20 10.2519/jospt.2003.33.1.4 12570282 (Pubitemid 36869653)
45. A method for assessing the quality of a randomized control trial. Chalmers TC, Smith H Jr, Blackburn B, Control Clin Trials 1981 2 1 31 49 10.1016/0197-2456(81)90056-8 7261638 (Pubitemid 11083624)
46. Aid to the evaluation of therapeutic studies. Reisch JS, Tyson JE, Mize SG, Pediatrics 1989 84 5 815 827 2797977 (Pubitemid 19265153)
47. Method for assessment of the reporting standard of clinical trials with roentgen contrast media. Andrew E, Acta Radiologica-Series Diagnosis 1984 25 1 55 58 (Pubitemid 14110411)
48. Do corticosteroids reduce mortality from alcoholic hepatitis? A meta-analysis of the randomized trials. Imperiale TF, McCullough AJ, Ann Intern Med 1990 113 4 299 307 10.7326/0003-4819-113-4-299 2142869 (Pubitemid 20254358)
49. Incorporating variations in the quality of individual randomized trials into meta-analysis. Detsky AS, Naylor CD, O'Rourke K, McGeer AJ, L'Abbe KA, J Clin Epidemiol 1992 45 3 255 265 10.1016/0895-4356(92)90085-2 1569422
50. Instruments for assessing the quality of drug studies published in the medical literature. Cho MK, Bero LA, JAMA 1994 272 2 101 104 10.1001/jama.1994.03520020027007 8015115 (Pubitemid 24206037)
51. Methods of randomized controlled clinical trials in health services research. Balas EA, Austin SM, Ewigman BG, Brown GD, Mitchell JA, Med Care 1995 33 7 687 299 10.1097/00005650-199507000-00005 7596208
52. Development of a tool to rate the quality assessment of randomized controlled trials using a Delphi technique. Sindhu F, Carpenter L, Seers K, J Adv Nurs 1997 25 6 1262 1268 10.1046/j.1365-2648.1997.19970251262.x 9181425 (Pubitemid 127452295)
53. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. Downs SH, Black N, J Epidemiol Commun Health 1998 52 6 377 384 10.1136/jech.52.6.377 (Pubitemid 28438523)
54. A systematic review of the relationship between overjet size and traumatic dental injuries. Nguyen QV, Bezemer PD, Habets L, Prahl-Andersen B, Eur J Orthodont 1999 21 5 503 515 10.1093/ejo/21.5.503
55. Teasing apart quality and validity in systematic reviews: an example from acupuncture trials in chronic neck and back pain. Smith LA, Oldman AD, McQuay HJ, Moore RA, Pain 2000 86 1-2 119 132 10779669 (Pubitemid 30211711)
56. A scale of methodological quality for clinical studies of radiologic examinations. Arrive L, Renard R, Carrat F, Belkacem A, Dahan H, Le Hir P, Monnier-Cholley L, Tubiana JM, Radiology 2000 217 1 69 74 11012425 (Pubitemid 33097580)
57. Quality of reporting of randomized trials as a measure of methodologic quality. Huwiler-Muntener K, Juni P, Junker C, Egger M, JAMA 2002 287 21 2801 2804 10.1001/jama.287.21.2801 12038917 (Pubitemid 34591968)
58. A scale for rating the quality of psychological trials for pain. Yates SL, Morley S, Eccleston C, Williams ACDC, Pain 2005 117 3 314 325 10.1016/j.pain.2005.06.018 16154704 (Pubitemid 41383527)
59. Reliability of three assessment tools used to evaluate randomized controlled trials for treatment of neck pain. Graham N, Haines T, Goldsmith CH, Gross A, Burnie S, Shahzad U, Talovikova E, Spine 2012 37 6 515 522 10.1097/BRS.0b013e31822671eb 21673624
60. Applying the Risk of Bias tool in a systematic review of combination longacting betaagonists and inhaled corticosteroids for persistent asthma. Hartling L, Bond K, Vandermeer B, Seida J, Dryden D, Rowe B, PLoS Med 2011 6 2 17242
61. Assessment of study quality for systematic reviews: a comparison of the Cochrane Collaboration Risk of Bias Tool and the Effective Public Health Practice Project Quality Assessment Tool: methodological research. Armijo-Olivo S, Stiles CR, Hagen NA, Biondo PD, Cummings GG, J Eval Clin Pract 2010 1 7
62. Testing the Risk of Bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, Tsertsvadze A, Hempel S, Shekelle P, Dryden DM, J Clin Epidemiol 2012 S0895-4356(12)00217-X. doi: 10.1016/j.jclinepi.2012.07.005
63. The hazards of scoring the quality of clinical trials for meta-analysis. Juni P, Witschi A, Bloch R, Egger M, JAMA 1999 282 11 1054 1060 10.1001/jama.282.11.1054 10493204 (Pubitemid 29436453)
64. Bad reporting does not mean bad methods for randomised trials: Observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, Djulbegovic B, BMJ 2004 328 7430 22 24 10.1136/bmj.328.7430.22 14703540 (Pubitemid 38067738)
65. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Sanderson S, Tatt ID, Higgins JPT, Int J Epidemiol 2007 36 3 666 676 10.1093/ije/dym018 17470488 (Pubitemid 47244573)