An Approach to Assess Generalizability in Comparative Effectiveness Research:A Case Study of the Whole Systems Demonstrator Cluster Randomized Trial Comparing Telehealth with Usual Care for Patients with Chronic Health Condition

Medical Decision Making

Volumn 35, Issue 8, 2015, Pages 1023-1036

  Steventon, Adam ^a,b   Grieve, Richard ^b   Bardsley, Martin ^c  

^a Health Foundation   (United Kingdom)

^b LONDON SCHOOL OF HYGIENE AND TROPICAL MEDICINE   (United Kingdom)

^c Health Education England   (United Kingdom)

Author keywords

causal inference; chronic health conditions; external validity; generalizability; randomized trials; telehealth

Indexed keywords

PLACEBO;

CHRONIC DISEASE; CLUSTER ANALYSIS; COMPARATIVE EFFECTIVENESS; CONTROLLED STUDY; ELECTRONIC HEALTH RECORD; ENGLAND; EPIDEMIOLOGY; HOSPITAL EMERGENCY SERVICE; HUMAN; MORTALITY; PRIMARY HEALTH CARE; PROCEDURES; RANDOMIZED CONTROLLED TRIAL; TELEMEDICINE; TREATMENT OUTCOME; UTILIZATION;

CHRONIC DISEASE; CLUSTER ANALYSIS; COMPARATIVE EFFECTIVENESS RESEARCH; ELECTRONIC HEALTH RECORDS; EMERGENCY SERVICE, HOSPITAL; ENGLAND; HUMANS; PLACEBOS; PRIMARY HEALTH CARE; TELEMEDICINE; TREATMENT OUTCOME;

EID: 84942924798     PISSN: 0272989X     EISSN: 1552681X     Source Type: Journal    
DOI: 10.1177/0272989X15585131     Document Type: Article

References (67)

1. Sculpher MJ, Pang FS, Manca A, et al. Generalisability in economic evaluation studies in healthcare: a review and case studies. Health Technol Assess (Rockv). 2004; 8 (49):iii-iv, 1-192.
2. Roland M, Torgerson DJ,. Understanding controlled trials: What are pragmatic trials? BMJ. 1998; 316 (7127): 285.
3. Gheorghe A, Roberts TE, Ives JC, Fletcher BR, Calvert M,. Centre selection for clinical trials and the generalisability of results: a mixed methods study. PLoS One. 2013; 8 (2): e56560.
4. Rothwell PM,. External validity of randomised controlled trials: "to whom do the results of this trial apply?". Lancet. 2005; 365 (9453): 82-93.
5. Gross CP, Krumholz HM, Van Wye G, Emanuel EJ, Wendler D,. Does random treatment assignment cause harm to research participants? PLoS Med. 2006; 3 (6): e188.
6. Cole SR, Stuart EA,. Generalizing evidence from randomized clinical trials to target populations: the ACTG 320 trial. Am J Epidemiol. 2010; 172 (1): 107-15.
7. Rosenbaum PR, Rubin DB,. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983; 70 (1): 41-55.
8. Sekhon JS, Grieve RD,. A matching method for improving covariate balance in cost-effectiveness analyses. Health Econ. 2012; 21 (6): 695-714.
9. Kolata G,. Method of study is criticized in groups health policy tests. New York Times. 3 2014: A1.
10. Olschewski M, Scheurlen H,. Comprehensive Cohort Study: an alternative to randomized consent design in a breast preservation trial. Methods Inf Med. 1985 J; 24 (3): 131-4.
11. Imai K, King G, Stuart EA,. Misunderstandings between experimentalists and observationalists about causal inference. J R Stat Soc Ser A Stat Soc. 2008; 171 (2): 481-502.
12. Stuart EA, Cole SR, Bradshaw CP, Leaf PJ,. The use of propensity scores to assess the generalizability of results from randomized trials. J R Stat Soc Ser A Stat Soc. 2011; 174 (2): 369-86.
13. Frangakis CE,. The calibration of treatment effects from clinical trials to target populations. Clin Trials. 2009; 6 (2): 136-40.
14. Hartman E, Grieve R, Ramsahai R, Sekhon JS,. From SATE to PATT: combining experimental with observational studies to estimate population treatment effects. J R Stat Soc Ser A Stat Soc. Epub ahead of print 2015. DOI: 10.1111/rssa.12094.
15. Pressler TR, Kaizar EE,. The use of propensity scores and observational data to estimate randomized controlled trial generalizability bias. Stat Med. 2013; 32 (20): 3552-68.
16. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M,. Developing and Evaluating Complex Interventions: New Guidance. London: Medical Research Council; 2008.
17. Fifield J, Forrest DD, Martin-Peele M, et al. A randomized, controlled trial of implementing the patient-centered medical home model in solo and small practices. J Gen Intern Med. 2013; 28 (6): 770-7.
18. Chaudhry SI, Mattera JA, Curtis JP, et al. Telemonitoring in patients with heart failure. N Engl J Med. 2010; 363 (24): 2301-9.
19. Wennberg DE, Marr A, Lang L, OMalley S, Bennett G,. A randomized trial of a telephone care-management strategy. N Engl J Med. 2010; 363 (13): 1245-55.
20. Hendy J, Chrysanthaki T, Barlow J, et al. An organisational analysis of the implementation of telecare and telehealth: the whole systems demonstrator. BMC Health Serv Res. 2012; 12: 403.
21. Cook TD, Campbell DT,. Quasi-experimentation: Design and Analysis Issues for Field Settings. Boston: Houghton Mifflin; 1979.
22. McCambridge J, Kypri K, Elbourne D,. In randomization we trust? There are overlooked problems in experimenting with people in behavioral intervention trials. J Clin Epidemiol. 2014; 67 (3): 247-53.
23. Bower P, Cartwright M, Hirani SP, et al. A comprehensive evaluation of the impact of telemonitoring in patients with long-term conditions and social care needs: protocol for the whole systems demonstrator cluster randomised trial. BMC Health Serv Res. 2011; 11: 184.
24. Steventon A, Bardsley M, Billings J, et al. Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trial. BMJ. 2012; 344: e3874.
25. Department of Health. Whole System Demonstrator Programme: Headline Findings-December 2011. London: Department of Health; 2011.
26. Department of Health. Our Health, Our Care, Our Say: A New Direction for Community Services. Cm 6737. London: The Stationery Office; 2006.
27. Bardsley M, Steventon A, Doll H,. Impact of telehealth on general practice contacts: findings from the Whole Systems Demonstrator cluster randomised trial. BMC Health Serv Res. 2013; 13: 395.
28. Health & Social Care Information Centre. Secondary Uses Service (SUS). Available from: URL: http://www.hscic.gov.uk/sus
29. Puffer S, Torgerson DJ, Watson J,. Evidence for risk of bias in cluster randomised trials: review of recent trials published in three general medical journals. BMJ. 2003; 327: 785-9.
30. Subramanian U, Hopp F, Lowery J, Woodbridge P, Smith D,. Research in home-care telemedicine: challenges in patient recruitment. Telemed J E Health. 2004; 10 (2): 155-61.
31. Sanders C, Rogers A, Bowen R, et al. Exploring barriers to participation and adoption of telehealth and telecare within the Whole System Demonstrator trial: a qualitative study. BMC Health Serv Res. 2012; 12 (1): 220.
32. Hopkinson NS,. Re: Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trial. BMJ. 2012; 344: e3874.
33. McCambridge J, Sorhaindo A, Quirk A, Nanchahal K,. Patient preferences and performance bias in a weight loss trial with a usual care arm. Patient Educ Couns. 2014; 95 (2): 243-7.
34. Ward E, King M, Lloyd M, Bower P, Friedl K,. Conducting randomized trials in general practice: methodological and practical issues. Br J Gen Pract. 1999; 49: 919-22.
35. De Craen AJM, Roos PJ, de Vries AL, Kleijnen J,. Effect of colour of drugs: systematic review of perceived effect of drugs and of their effectiveness. BMJ. 1996; 313: 1624-6.
36. Hobbs S,. Newham WSD TeleHealth: improving quality of life at home. Available from: URL: http://www.newhamwsdtrial.org/user-files/downloads/PR/PRJuly-THTerryMunro.pdf
37. Car J, Huckvale K, Hermens H,. Telehealth for long term conditions. BMJ. 2012; 344: e4201.
38. McLean S, Sheikh A, Cresswell K, et al. The impact of telehealthcare on the quality and safety of care: a systematic overview. PLoS One. 2013; 8 (8): e71238.
39. McLean S, Nurmatov U, Liu JLY, Pagliari C, Car J, Sheikh A,. Telehealthcare for chronic obstructive pulmonary disease. Cochrane Database Syst Rev. 2011;(7): CD007718.
40. Polisena J, Tran K, Cimon K, et al. Home telemonitoring for congestive heart failure: a systematic review and meta-analysis. J Telemed Telecare. 2010; 16 (2): 68-76.
41. Polisena J, Tran K, Cimon K, Hutton B, McGill S, Palmer K,. Home telehealth for diabetes management: a systematic review and meta-analysis. Diabetes Obes Metab. 2009; 11 (10): 913-30.
42. Hauck K, Zhao X,. How dangerous is a day in hospital? A model of adverse events and length of stay for medical inpatients. Med Care. 2011; 49 (12): 1068-75.
43. Robinson S,. NHS England abandons health secretarys pledge on telehealth. Available from: URL: http://www.gponline.com/bulletin/daily-news/article/1222218/nhs-england-abandons-health-secretarys-pledge-telehealth/?DCMP=EMC-ED-News,jobsandCPD-1222218
44. Takahashi PY, Pecina JL, Upatising B, et al. A randomized controlled trial of telemonitoring in older adults with multiple health issues to prevent hospitalizations and emergency department visits. JAMA Intern Med. 2012; 172 (10): 773-9.
45. Hartman E, Grieve R, Sekhon JS,. From SATE to PATT: The Essential Role of Placebo Tests for Combining Experimental with Observational Studies. Berkeley: University of California, Berkeley; 2010.
46. Rubin DB,. For objective causal inference, design trumps analysis. Ann Appl Stat. 2008; 2 (3): 808-40.
47. Jones AM,. Identification of treatment effects in Health Economics. Health Econ. 2007; 16 (11): 1127-31.
48. Hartman E, Hidalgo FD,. Whats the Alternative? An Equivalence Approach to Balance and Placebo Tests. Berkeley: University of California, Berkeley; 2011.
49. Billings J, Dixon J, Mijanovich T, Wennberg D,. Case finding for patients at risk of readmission to hospital: development of algorithm to identify high risk patients. BMJ. 2006; 333 (7563): 327.
50. Wennberg D, Siegel M, Darin B, et al. Combined Predictive Model: Final Report and Technical Documentation. London: Kings Fund; 2006.
51. Austin PC,. A critical appraisal of propensity-score matching in the medical literature between 1996 and 2003. Stat Med. 2008; 27: 2037-49.
52. Normand S-LT, Landrum MB, Guadagnoli E, et al. Validating recommendations for coronary angiography following acute myocardial infarction in the elderly: a matched analysis using propensity scores. J Clin Epidemiol. 2001; 54 (4): 387-98.
53. Austin PC,. Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Stat Med. 2009; 28: 3083-107.
54. Ho DE, Imai K, King G, Stuart EA,. Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Polit Anal. 2007; 15 (3): 199-236.
55. Van Walraven C, Austin P,. Administrative database research has unique characteristics that can risk biased results. J Clin Epidemiol. 2012; 65 (2): 126-31.
56. Steventon A, Grieve R, Sekhon JS,. A comparison of alternative strategies for choosing control populations in observational studies. Health Serv Outcomes Res Methodol. Epub ahead of print 2015. DOI: 10.1007/s10742-014-0135-8.
57. Govan L, Wu O, Briggs A, et al. Achieved levels of HbA1c and likelihood of hospital admission in people with type 1 diabetes in the Scottish population: a study from the Scottish Diabetes Research Network Epidemiology Group. Diabetes Care. 2011; 34 (9): 1992-7.
58. Xu W, Collet J-P, Shapiro S, et al. Independent effect of depression and anxiety on chronic obstructive pulmonary disease exacerbations and hospitalizations. Am J Respir Crit Care Med. 2008; 178 (9): 913-20.
59. Quintana JM, Esteban C, Unzurrunzaga A, et al. Predictive score for mortality in patients with COPD exacerbations attending hospital emergency departments. BMC Med. 2014; 12: 66.
60. Lee D, Austin P, Rouleau J,. Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model. JAMA. 2003; 290 (19): 2581-7.
61. Rubin DB,. On the limitations of comparative effectiveness research. Stat Med. 2010; 29 (19): 1991-5.
62. Olschewski M, Schumacher M, Davis KB,. Analysis of randomized and nonrandomized patients in clinical trials using the comprehensive cohort follow-up study design. Control Clin Trials. 1992; 13 (3): 226-39.
63. Brewin CR, Bradley C,. Patient preferences and randomised clinical trials. BMJ. 1989; 299 (6694): 313-5.
64. Van Staa T-P, Goldacre B, Gulliford M, et al. Pragmatic randomised trials using routine electronic health records: putting them to the test. BMJ. 2012; 344: e55.
65. Begg C, Cho M, Eastwood S, et al. Improving the quality of randomized controlled trials: the CONSORT statement. JAMA. 1966; 276 (8): 637-9.
66. Cartwright M, Hirani SP, Rixon L, et al. Effect of telehealth on quality of life and psychological outcomes over 12 months (Whole Systems Demonstrator telehealth questionnaire study): nested study of patient reported outcomes in a pragmatic, cluster randomised controlled trial. BMJ. 2013; 346: f653.
67. LaLonde R,. Evaluating the econometric evaluations of training programs with experimental data. Am Econ Rev. 1986; 76 (4): 604-20.