Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in Physical Therapy trials

PLoS ONE

Volumn 9, Issue 5, 2014, Pages

  Armijo Olivo, Susan ^a   Ospina, Maria ^a   Da Costa, Bruno R ^b   Egger, Matthias ^c   Saltaji, Humam ^a   Fuentes, Jorge ^a,d   Ha, Christine ^a   Cummings, Greta G ^a  

^a UNIVERSITY OF ALBERTA   (Canada)

^b FLORIDA INTERNATIONAL UNIVERSITY   (United States)

^c UNIVERSITY OF BERN   (Switzerland)

^d UNIVERSIDAD CATÓLICA DEL MAULE   (Chile)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; BLINDED EXTERNAL REVIEW; COCHRANE LIBRARY; COMPARATIVE STUDY; HUMAN; INTERRATER RELIABILITY; META ANALYSIS; PEER REVIEW; PHYSIOTHERAPY; RANDOMIZED CONTROLLED TRIAL (TOPIC); RISK ASSESSMENT; RISK OF BIAS; SYSTEMATIC REVIEW; LITERATURE; METHODOLOGY; REPRODUCIBILITY; STATISTICAL BIAS;

BIAS (EPIDEMIOLOGY); HUMANS; PHYSICAL THERAPY MODALITIES; RANDOMIZED CONTROLLED TRIALS AS TOPIC; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; REVIEW LITERATURE AS TOPIC;

EID: 84901260203     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0096920     Document Type: Article

References (54)

1. Armijo-Olivo S, Macedo LG, Gadotti IC, Fuentes J, Stanton T, et al. (2008) Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review. Physical Therapy 88: 156-175. (Pubitemid 351210558)
2. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, et al. (1995) Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Controlled Clinical Trials 16: 62-73.
3. Higgins J, Altman DG (2008) Chapter 8: Assessing risk of bias in included studies. In: Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 500 [updated February 2008] version 5.0 ed: Available from www.cochrane-handbook.org, February, 2008.
4. Lundh A, Gotzsche PC (2008) Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies. BMC Medical Research Methodology 8.
5. Schulz KF, Chalmers I, Hayes RJ, Altman DG (1995) Empirical evidence of bias: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. Journal of the American Medical Association 273: 408-412.
6. Emerson JD, Burdick E, Hoaglin DC, Mosteller F, Chalmers TC (1990) An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. Controlled Clinical Trials 11: 339-352.
7. Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, et al. (1999) Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. Health Technology Assessment (Winchester, England) 3: i-iv.
8. Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, et al. (2012) Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Annals of Internal Medicine 157: 429-438.
9. Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, et al. (2009) Risk of bias versus quality assessment of randomised controlled trials: Cross sectional study. BMJ 339: 1017.
10. Armijo-Olivo S, Stiles CR, Hagen NA, Biondo PD, Cummings GG (2012) Assessment of study quality for systematic reviews: A comparison of the Cochrane Collaboration Risk of Bias Tool and the Effective Public Health Practice Project Quality Assessment Tool: Methodological research. Journal of Evaluation in Clinical Practice 18: 12-18.
11. Boutron I, Ravaud P (2012) Classification systems to improve assessment of risk of bias. Journal of Clinical Epidemiology 65: 236-238.
12. Hartling L, Bond K, Vandermeer B, Seida J, Dryden D, et al. (2011) Applying the Risk of Bias tool in a systematic review of combination longacting betaagonists and inhaled corticosteroids for persistent asthma. PLoS Medicine 6.
13. Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, et al. (2012) Testing the Risk of Bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. Journal of Clinical Epidemiology 66: 973-981.
14. World Confederation for Physical Therapy (2011) Position statement: standards of physical therapy practice. World Confederation for Physical Therapy. 1-45 p.
15. Higgins JPT, Altman DG, Goetzsche PC, Juni P, Moher D, et al. (2011) The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 343.
16. Furlan AD, Pennick V, Bombardier C, Van Tulder M (2009) 2009 Updated method guidelines for systematic reviews in the cochrane back review group. Spine 34: 1929-1941.
17. Higgins J, Altman D (2008) Chapter 8: Assessing risk of bias in included studies In: Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions version 50. Chichester, UK: John Wiley & Sons, Ltd.
18. Schuck P (2004) Assessing Reproducibility for Interval Data in Health-Related Quality of Life Questionnaires: Which Coefficient Should Be Used? Quality of Life Research 13: 571-586.
19. Cohen J (1968) Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin 70: 213-220.
20. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33: 159-174. (Pubitemid 8054849)
21. Byrt T (1996) How good is that agreement? Epidemiology (Cambridge, Mass) 7: 561.
22. Fransen M, McConnell S, Hernandez MG, Reichenbach S (2009) Exercise for osteoarthritis of the hip. Cochrane Database of Systematic Reviews: Reviews 2009, Issue 3 JohnWiley& Sons, Ltd Chichester, UK DOI: 101002/14651858CD007.
23. Handoll-Helen HG, Cameron ID, Mak-Jenson CS, Finnegan TP (2009) Multidisciplinary rehabilitation for older people with hip fractures. Cochrane Database of Systematic Reviews Reviews 2009, Issue 4 JohnWiley& Sons, LtdChichester, UK DOI: 101002.
24. Harvey LA, Brosseau L, Herbert RD (2010) Continuous passive motion following total knee arthroplasty in people with arthritis. Cochrane Database of Systematic Reviews Reviews 2010, Issue 3 JohnWiley& Sons, LtdChichester, UK DOI: 101002/14.
25. Katalinic OM, Harvey LA, Herbert RD, Moseley AM, Lannin NA, et al. (2010) Stretch for the treatment and prevention of contractures. Cochrane Database of Systematic Reviews Reviews 2010, Issue 9 JohnWiley& Sons, LtdChichester, UK DOI: 101002/14 2010.
26. Manheimer E, Cheng K, Linde K, Lao L, Yoo J, et al. (2010) Acupuncture for peripheral joint osteoarthritis. Cochrane Database of Systematic Reviews: Reviews 2010 Issue 1 JohnWiley& Sons, Ltd Chichester, UK.
27. Ostelo-Raymond WJG, Costa-Leonardo OP, Maher CG, de-Vet-Henrica CW, van-Tulder MW (2008) Rehabilitation after lumbar disc surgery. Cochrane Database of Systematic Reviews Reviews 2008, Issue 4 JohnWiley& Sons, LtdChichester, UK.
28. Rutjes-Anne WS, Nüesch E, Sterchi R, Jüni P (2010) Therapeutic ultrasound for osteoarthritis of the knee or hip. Cochrane Database of Systematic Reviews Reviews 2010, Issue 1 JohnWiley & Sons, LtdChichester, UK DOI: 101002/14651858CD00.
29. Rutjes-Anne WS, Nüesch E, Sterchi R, Kalichman L, Hendriks E, et al. (2009) Transcutaneous electrostimulation for osteoarthritis of the knee. Cochrane Database of Systematic Reviews 2009: Issue;4 JohnWiley & Sons, Ltd Chichester CD002823.
30. Schaafsma F, Schonstein E, Whelan KM, Ulvestad E, Kenny DT, et al. (2010) Physical conditioning programs for improving work outcomes in workers with back pain. Cochrane Database of Systematic Reviews Reviews 2010, Issue 1 John Wiley & Sons, Ltd Chichester, UK.
31. Davies P, Taylor F, Beswick A, Wise F, Moxham T, et al. (2010) Promoting patient uptake and adherence in cardiac rehabilitation. Cochrane Database of Systematic Reviews Reviews 2010, Issue 7 John Wiley& Sons Ltd Chichester, UK.
32. Effing T, Monninkhof-Evelyn EM, Valk-Paul PDLP, Zielhuis-Gerhard GA, Walters EH, et al. (2007) Self-management education for patients with chronic obstructive pulmonary disease. Cochrane Database of Systematic Reviews Reviews 2007, Issue 4 John Wiley& Sons Ltd Chichester, UK.
33. Puhan MA, Gimeno SE, Scharplatz M, Troosters T, Walters EH, et al. (2009) Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease. Cochrane Database of Systematic Reviews Reviews 2009, Issue 1 John Wiley& Sons Ltd Chichester, UK.
34. Taylor RS, Dalal H, Jolly K, Moxham T, Zawada A (2010) Home-based versus centre-based cardiac rehabilitation. Cochrane Database of Systematic Reviews Reviews 2010, Issue 1 John Wiley& Sons, Ltd Chichester, UK. DOI: 101002/14651858CD.
35. Sirtori V, Corbetta D, Moja L, Gatti R (2009) Constraint-induced movement therapy for upper extremities in stroke patients. Cochrane Database of Systematic Reviews Reviews 2009, Issue 4 John Wiley& Sons, Ltd Chichester, UK.
36. States RA, Pappas E, Salem Y (2009) Overground physical therapy gait training for chronic stroke patients with mobility deficits. Cochrane Database of Systematic Reviews Reviews 2009, Issue 3 JohnWiley& Sons, LtdChichester, UK
37. Kramer MS, McDonald SW (2006) Aerobic exercise for women during pregnancy. Kramer MichaelS, McDonald Sheila W Aerobic exercise for women during pregnancy Cochrane Database of Systematic Reviews: Reviews 2006 Issue 3 John Wiley& Sons, Ltd Chichester, UK. DOI: 101002/14651858CD000180pub2.
38. Orozco LJ, Buchleitner AM, Gimenez PG, Figuls M, Richter B, et al. (2008) Exercise or exercise and diet for preventing type 2 diabetes mellitus. Cochrane Database of Systematic Reviews Reviews 2008, Issue 3 John Wiley& Sons Ltd Chichester, UK.
39. Egger M, Juni P, Bartlett C, Holenstein F, Sterne J (2003) How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health technology assessment 7: 1-76.
40. Wood L, Egger M, Gluud LL, Schulz KF, Juni P, et al. (2008) Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: Meta-epidemiological study. BMJ 336: 601-605. (Pubitemid 351439518)
41. Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, et al. (2007) Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. International Journal of Epidemiology 36: 847-857. (Pubitemid 47394153)
42. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, et al. (1998) Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?[see comment]. Lancet 352: 609-613. (Pubitemid 28380142)
43. Villari P, Manzoli L, Boccia A (2004) Methodological quality of studies and patient age as major sources of variation in efficacy estimates of influenza vaccination in healthy adults: A meta-analysis. Vaccine 22: 3475-3486. (Pubitemid 39078767)
44. Manzoli L, Schioppa F, Boccia A, Villari P (2007) The efficacy of influenza vaccine for healthy children: A meta-analysis evaluating potential sources of variation in efficacy estimates including study quality. Pediatric Infectious Disease Journal 26: 97-106. (Pubitemid 46183358)
45. Nuesch E, Trelle S, Reichenbach S, Rutjes AWS, Burgi E, et al. (2009) The effects of excluding patients from the analysis in randomised controlled trials: Meta-epidemiological study. BMJ 339: 679-683.
46. Abraha I, Montedori A (2010) Modified intention to treat reporting in randomised controlled trials: Systematic review. BMJ (Online) 341: 33.
47. Armijo-Olivo S, Warren S, Magee D (2009) Intention to treat analysis, compliance, drop-outs and how to deal with missing data in clinical research: a review. Physical Therapy Reviews 14: 36-49.
48. Hewitt CE, Kumaravel B, Dumville JC, Torgerson DJ (2010) Assessing the impact of attrition in randomized controlled trials. Journal of Clinical Epidemiology 63: 1264-1270.
49. Kjaergard LL, Als-Nielsen B (2002) Association between competing interests and authors' conclusions: Epidemiological study of randomised clinical trials published in the BMJ. British Medical Journal 325: 249-252. (Pubitemid 34864391)
50. Trowman R, Dumville JC, Torgerson DJ, Cranny G (2007) The impact of trial baseline imbalances should be considered in systematic reviews: a methodological case study. Journal of Clinical Epidemiology 60: 1229-1233. (Pubitemid 350061144)
51. Graham N, Haines T, Goldsmith CH, Gross A, Burnie S, et al. (2011) Reliability of three assessment tools used to evaluate randomized controlled trials for treatment of neck pain. Spine.
52. Unnebrink K, Windeler J (2001) Intention-to-treat: Methods for dealing with missing values in clinical trials of progressively deteriorating diseases. Statistics in Medicine 20: 3931-3946.
53. Wright CC, Sim J (2003) Intention-to-treat approach to data from randomized controlled trials: A sensitivity analysis. Journal of Clinical Epidemiology 56: 833-842. (Pubitemid 37162657)
54. Herbison P, Hay-Smith J, Gillespie WJ (2006) Adjustment of meta-analyses on the basis of quality scores should be abandoned. Journal of Clinical Epidemiology 59: 1249.e1241-1249.e1211.