The case for randomized controlled trials to assess the impact of clinical information systems

Journal of the American Medical Informatics Association

Volumn 18, Issue 2, 2011, Pages 173-180

  Liu, Joseph L Y ^a,b   Wyatt, Jeremy C ^a,c  

^a UNIVERSITY OF DUNDEE   (United Kingdom)

^b UNIVERSITY OF EDINBURGH   (United Kingdom)

^c UNIVERSITY OF WARWICK   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; CLINICAL EVALUATION; COMPUTER SYSTEM; COST EFFECTIVENESS ANALYSIS; DECISION SUPPORT SYSTEM; EVIDENCE BASED PRACTICE; HUMAN; MEDICAL INFORMATION SYSTEM; OUTCOME ASSESSMENT; RANDOMIZED CONTROLLED TRIAL (TOPIC); RELIABILITY; RESEARCH ETHICS;

EVIDENCE-BASED PRACTICE; HEALTH SERVICES RESEARCH; HUMANS; INFORMATION SYSTEMS; RANDOMIZED CONTROLLED TRIALS AS TOPIC; RESEARCH DESIGN;

EID: 79953093531     PISSN: 10675027     EISSN: 1527974X     Source Type: Journal    
DOI: 10.1136/jamia.2010.010306     Document Type: Article

References (115)

1. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005;293:1197-203.
2. Tierney WM, Overhage JM, McDonald CJ. A plea for controlled trials in medical informatics. J Am Med Inform Assoc 1994;1:353-5.
3. Heathfield HA, Buchan IB. Current evaluations of information technology in health care are often inadequate. BMJ 1996;313:1008.
4. Heathfield HA, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ 1998;316:1959-61.
5. Kaplan B. Evaluating informatics applicationsdsome alternative approaches, and call for methodological pluralism. Int J Med Inform 2001;64:39-56.
6. UK Institute of Health Informatics. Evaluation of Electronic Patient and Health Record Projects. Winchester: ERDIP Programme, NHS Information Authority, 2001.
7. Stoop A, Heathfield H, de Mul M, et al. Evaluation of patient care information systems: theory and practice. In: Berg M, ed. Health Information Management: Integrating Information Technology in Health Care Work. London: Routledge, 2004:206-29.
8. Brender J. Handbook of Evaluation Methods for Health Informatics. London: Elsevier Academic Press, 2006.
9. Catwell L, Sheikh A. Evaluating eHealth interventions: the need for continuous systematic evaluation. PLoS Med 2009;6:e10000126.
10. Rigby M. Health informatics as a tool to improve quality in non-acute carednew opportunities and a matching need for a new evaluation paradigm. Int J Med Inform 1999;56:141-50.
11. Patel VL, Kaufmann DR, Arocha JF. Emerging paradigms of cognition in medical decision making. J Biomed Inform 2002;35:52-75.
12. Kaplan B, Shaw NT. People, Organizational, and Social Issues. IMIA Yearbook of Medical Informatics. Heidelberg: Schattauer, 2002:91-102.
13. Beale S, Sanderson D, Kruger J; for the York Health Economics Consortium. Evaluation of the Telecare Development Programme: Final Report. Edinburgh: Scottish Government, 2009.
14. Friedman CP, Wyatt JC. Evaluation Methods in Bio-Medical Informatics. 2nd edn. New York: Springer-Verlag, 2006.
15. Altman DG. Practical Statistics for Medical Research. London: Chapman & Hall, 1991.
16. Pope C, Mays N. Qualitative Research in Health Care. 2nd edn. London: BMJ Publishing, 2000.
17. Rothman KJ, Greenland S. Modern Epidemiology. 2nd edn. Philadelphia: Lippincott-Raven, 1998.
18. Harris AD, McGregor JC, Perencevich EN, et al. The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc 2006;13:16-23.
19. de Keizer NF, Ammenwerth E. The quality of evidence in health informatics: how did the quality of healthcare IT evaluation publications develop from 1982 to 2005? Int J Med Inform 2008;77:41-9.
20. Schedlbauer A, Prasad V, Mulvaney C, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior? J Am Med Inform Assoc 2009;16:531-8.
21. Liu JLY, Wyatt JC, Deeks JJ, et al. Systematic reviews of clinical decision tools for acute abdominal pain. Health Technol Assess 2006;10:1-186, iiieiv.
22. Copi IM, Cohen C. Introduction to Logic. 9th edn. Englewood Cliffs, NJ: Prentice-Hall, 1994.
23. Wyatt JC, Wyatt SM. When and how to evaluate health information systems? Int J Med Inform 2003;69:251-9.
24. Pope C, Mays N. Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services. BMJ 1995;311:42-5.
25. Drummond MF, Sculpher MJ, Torrance GW, et al. Methods for the Economic Evaluation of Health Care Programmes. 3rd edn. New York: Oxford University Press, 2005.
26. Pocock SJ. Clinical Trials: A Practical Approach. Chichester: John Wiley and Sons, 1983.
27. Schulz KF, Altman DG, Moher D; the CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel-group randomized trials. BMJ 2010;340:c332.
28. Moher D, Hopewell S, Schulz KF, et al. The CONSORT Statement 2010 updated guidelines for reporting randomized trials: explanation and elaboration. BMJ 2010;340:c869.
29. Friedman CP, Elstein AS, Wolf FM, et al. Enhancement of clinicians' diagnostic reasoning by computer-based consultations: a multisite study of 2 systems. JAMA 1999;282:1851-6.
30. Macintyre S, Petticrew M. Good intentions and received wisdom are not enough. J Epidemiol Community Health 2000;54:802-3.
31. Wynder EL. Studies in mechanism and prevention: striking a proper balance. Am J Epidemiol 1994;139:547-9.
32. McLean S, Chandler D, Nurmatov U, et al. Telehealthcare for asthma. Cochrane Database Syst Rev 2010;(10):CD007717.
33. Liu JLY, Wyatt JC. The Impact of Telepsychiatry, Teledermatology, Teleent and Teleradiology on the Health Services: A Rapid Review. Portland, OR: Medicaid Evidence-Based Decisions Program, 2008.
34. Clark JS, Mair FS, O'Donnell C, et al. E-health: implementation and evaluation research in Scotland-a scoping exercise. J Telemed Telecare 2008;14:119-21.
35. Abraham EP, Chain E, Fletcher CM. Further observations on penicillin. Lancet 1941;2:177-88.
36. Peto R, Pike MC, Armitage P, et al. Design and analysis of randomized clinical trials requiring prolonged observation of each patient. I. Introduction and design. Br J Cancer 1976;34:585-612.
37. Collins R, Peto R, Gray R, et al. Large-Scale Randomized Evidence: Trials and Overviews. Oxford Textbook of Medicine. Oxford: Oxford University Press, 1998:21-32.
38. Peto R, Baigent C. Trials: the next 50 years. Large scale randomised evidence of moderate benefits. BMJ 1998;317:1170-1.
39. Collins R, MacMahon S. Reliable assessment of the effects of treatment on mortality and major morbidity, I: clinical trials. Lancet 2001;357:373-80.
40. Sacks H, Chalmers TC, Smith H Jr. Randomized versus historical controls for clinical trials. Am J Med 1982;72:233-40.
41. Tang JL, Leung PC. An efficacy-driven approach to the research and development of traditional Chinese medicine. Hong Kong Med J 2001;7:375-80.
42. Anon. Effectiveness of health checks conducted by nurses in primary care: final results of the OXCHECK study. Imperial Cancer Research Fund OXCHECK Study Group. BMJ 1995;310:1099-104.
43. Proudfoot J, Guest D, Carson J, et al. Effect of cognitive-behavioural training on job-finding among long-term unemployed people. Lancet 1997;350:96-100.
44. Wyatt JC, Paterson-Brown S, Johanson R, et al. Trial of outreach visits to enhance the use of systematic reviews in 25 obstetric units. BMJ 1998;317:1041-6.
45. Hobbs M, Mayou R, Harrison B, et al. A randomised controlled trial of psychological debriefing for victims of road traffic accidents. BMJ 1996;313:1438-9.
46. Allen K, Blascovich J. The value of service dogs for people with severe ambulatory disabilities. A randomised controlled trial. JAMA 1996;275:1001-6.
47. Byrd RC. Positive therapeutic effects of intercessory prayer in a coronary care unit population. South Med J 1988;81:826-9.
48. Moses S, Manji F, Bradley JE, et al. Impact of user fees on attendance at a referral centre for sexually transmitted diseases in Kenya. Lancet 1992;340:463-6.
49. Kelly JA, Murphy DA, Sikkema KJ, et al. Randomised, controlled, community-level HIV-prevention intervention for sexual-risk behaviour among homosexual men in US cities. Community HIV Prevention Research Collaborative. Lancet 1997;350:1500-5.
50. Close J, Ellis M, Hooper R, et al. Prevention of falls in the elderly trial (PROFET): a randomised controlled trial. Lancet 1999;353:93-7.
51. Frasure-Smith N, Lesperance F, Prince RH, et al. Randomised trial of home-based psychosocial nursing intervention for patients recovering from myocardial infarction. Lancet 1997;350:473-9.
52. Oakley A. Experimentation and social interventions: a forgotten but important history. BMJ 1998;317:1239-42.
53. Wellwood J, Johannesen S, Spiegelhalter DJ. How does computer aided diagnosis improve the managment of acute abdominal pain? Ann R Coll Surg Engl 1992;74:40-6.
54. Friedman LM, Furberg CD, DeMets DL. Fundamentals of Clinical Trials. New York: Springer Verlag, 1998.
55. Friedman CP, Abbas UL. Is medical informatics a mature science? A review of measurement practice in outcome studies of clinical systems. Int J Med Inform 2003;69:261-72.
56. Black A, Car J, Majeed A, et al. Strategic considerations for improving the quality of eHealth research: we need to improve the quality and capacity of academia to undertake informatics research. Inform Prim Care 2008;16:175-7.
57. Car J, Black A, Anandan C, et al. The Impact of Ehealth on the Quality and Safety of Healthcare: A Systematic Overview and Synthesis of the Literature. Birmingham: NHS Connecting for Health Evaluation Programme, 2008.
58. Hunt DL, Haynes RB, Hanna SE, et al. Effects of computer-based clinical decision support systems on physician perfomance and patient outcomes: a systematic review. JAMA 1998;280:1339-46.
59. Balas EA, Austin SM, Mitchel JA, et al. The clinical value of computerized information services. A review of 98 randomized clinical trials. Arch Fam Med 1996;5:271-8.
60. Johnston ME, Langton KB, Haynes RB, et al. Effects of computer-based clinical decision support systems on clinician performance and patient outcome: a critical appraisal of research. Ann Intern Med 1994;120:135-42.
61. Haynes RB, Walker CJ. Computer-aided quality assurance: a critical appraisal of research. Arch Intern Med 1987;147:1297-301.
62. Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005;293:1223-38.
63. Leung GM, Johnston JM, Tin KYT, et al. Randomised controlled trial of clinical decision tools to improve learning of evidence based medicine in medical students. BMJ 2003;327:1090.
64. Goud R, de Keizer NF, ter Riet G, et al. Effect of guideline based computerised decision support on decision making of multidisciplinary teams: cluster randomised trial in cardiac rehabiltation. BMJ 2009;338:b1440.
65. Swartz LHG, Noell JW, Schroeder SW, et al. A randomised control study of a fully automated internet based smoking cessation programme. Tob Control 2006;15:7-12.
66. Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet 2005;365:1159-62.
67. Tang JL. Research priorities in traditional Chinese medicine. BMJ 2006;333:391-4.
68. Deeks JJ, Dinnes J, D'Amico R, et al. Evaluating non-randomised intervention studies. Health Technol Assess 2003;7:iiiex, 1-173.
69. Adams ID, Chan M, Clifford PC, et al. Computer aided diagnosis of acute abdominal pain: a multicentre study. BMJ (Clin Res Ed) 1986;293:800-4.
70. Murray E, Burns J, See ST, et al. Interactive health communication applications for people with chronic disease. Cochrane Database Syst Rev 2005;(4):CD004274.
71. Clark RA, Inglis SC, McAlister FA, et al. Telemonitoring or structured telephone support programmes for patients with chronic heart failure: systematic review and meta-analysis. BMJ 2007;334:942-5.
72. Guyatt G, Mills EJ, Elbourne D. In the era of systematic reviews, does the size of an individual trial still matter? PLoS Med 2008;5:e4.
73. Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ 2010;10:12.
74. Greenhalgh T, Potts HW, Wong G, et al. Tensions and paradoxes in electronic patient record research: a systematic literature review using the meta-narrative method. Milbank Q 2009;87:729-88.
75. Cresswell KM, Worth A, Sheikh A. Actor-network theory and its role in understanding the implementation of information technology developments in healthcare. BMC Med Inform Decis Mak 2010;10:67.
76. Shepperd S, Lewin S, Straus S, et al. Can we systematically review studies that evaluate complex interventions? PLoS Med 2009;6:e10000086.
77. Pawson R. Evidence-based policy: the promise of 'realist synthesis.' Evaluation 2002;8:340-58.
78. Pawson R, Greenhalgh T, Harvey G, et al. Realist review-a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy 2005;(10 Suppl 1):21-34.
79. Greenhalgh T, Stones R. Theorising big IT programmes in healthcare: strong structuration theory meets actor-network theory. Soc Sci Med 2010;70:1285-94.
80. Edwards P, Arango M, Balica L, et al; CRASH trial collaborators. Final results of MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroids in adults with head injury-outcomes at 6 months. Lancet 2005;365:1957-9.
81. Sullivan FM, Swan IRC, Donnan PT, et al. Early treatment with prednisolone or acyclovir in Bell's Palsy. N Engl J Med 2007;357:1598-607.
82. Altman D, Carroli G, Duley L, et al; The Magpie Trial Collaborative Group. Do women with pre-eclampsia, and theirr babies, benefit from magnesium sulphate? The Magpie Trial: a randomised placebo-controlled trial. Lancet 2002;359:1877-90.
83. Hennekens CH, Buring JE, Manson JE, et al. Lack of effect of long-term supplementation with beta carotene on the incidence of malignant neoplasms and cardiovascular disease. N Engl J Med 1996;334:1145-9.
84. Karsh BT, Weinger MB, Abbott PA, et al. Health information technology: fallacies and sober realities. J Am Med Inform Assoc 2010;17:617-23.
85. Rigby M, Forsstrom J, Roberts R, et al. Verifying quality and safety in health informatics services. BMJ 2001;323:552-6.
86. Roberts JM, Copeland KL. Clinical websites are currently dangerous to health. Int J Med Inform 2001;62:181-7.
87. Donawa M. FDA final guidance on software validation. Med Device Technol 2002;13:20-4.
88. McDonald CJ. Computerization can create safety hazards: a bar-coding near miss. Ann Intern Med 2006;144:510-16.
89. Newton RC, Mytton OT, Aggarwal R, et al. Making existing technology safer in healthcare. Qual Saf Health Care 2010;19(Suppl 2):i15-24.
90. Robertson A, Cresswell K, Takian A, et al. Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation. BMJ 2010;341: c4564.
91. Ong SG, Liu J, Wong CM, et al. Studies on the respiratory health of primary school children in urban communities of Hong Kong. Sci Total Environ 1991;106:121-35.
92. Peters J, Hedley AJ, Wong CM, et al. Effects of an ambient air pollution intervention and environmental tobacco smoke on children's respiratory health in Hong Kong. Int J Epidemiol 1996;25:821-8.
93. Hedley AJ, Wong CM, Thach TQ, et al. Cardiorespiratory and all-cause mortality after restriction on sulphur content of fuel in Hong Kong: an intervention study. Lancet 2002;361:787-8.
94. Ho LM, McGhee SM, Hedley AJ, et al. The application of a computerized problem-oriented medical record system and its impact on patient care. Int J Med Inform 1999;55:47-59.
95. Contopoulos-Ioannidis DG, Ntzani EE, Ioannidis JP. Translation of highly promising basic science research into clinical applications. Am J Med 2003;114:477-84.
96. Crowley WF Jr. Translation of basic research into useful treatments: how often does it occur? Am J Med 2003;114:503-5.
97. Mill JS. System of Logic, Ratiocinative and Inductive. 5th edn. London: Parker, Son and Bowin, 1862.
98. Copi IM, Cohen C. Introduction to Logic. 13th edn. Englewood Cliffs, NJ: Prentice-Hall, 2008.
99. Torgerson DJ. Contamination in trials: is cluster randomisation the answer? BMJ 2001;322:355-7.
100. Puffer S, Torgerson DJ, Watson J. Evidence for risk of bias in cluster randomised trials: review of recent trials published in three general medical journals. BMJ 2003;327:785-9.
101. Gilbody S, Bower P, Torgerson D, et al. Cluster randomized trials produced similar results to individually randomized trials in a meta-analysis of enhanced care for depression. J Clin Epidemiol 2008;61:160-8.
102. Bennett S, Braunholtz D, Chinn S, et al. Cluster Randomised Trials: Methodological and Ethical Considerations. MRC Clinical Trials Series. London: Medical Research Council, 2002.
103. Eccles M, McColl E, Steen N, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ 2002;325:941-4.
104. Eminovic N, de Keizer NF, Wyatt JC, et al. Teledermatologic consultation and reduction in referrals to dermatologists: a cluster randomized controlled trial. Arch Dermatol 2009;145:558-64.
105. Friedman B. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:141-5.
106. Fries JF, Krishnan E. Equipoise, design bias and randomized controlled trials: the elusive ethics of new drug development. Arthritis Res Ther 2004;6:R250-5.
107. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol 2006;6:54.
108. Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials 2007;28:182-91.
109. Grimshaw J, Campbell M, Eccles M, et al. Experimental and quasi-experimental designs for evaluating guideline implementation strategies. Fam Pract 2000;(17 Suppl 1):S11-18.
110. Carey TS, Garrett J. Patterns of ordering diagnostic tests for patients with acute low back pain. The North Carolina Back Pain Project. Ann Intern Med 1996;125:807-14.
111. Jones TV, Gerrity MS, Earp J. Written case simulations: do they predict physicians' behaviour? J Clin Epidemiol 1990;43:805-15.
112. Schriger DL, Baraff LJ, Buller K, et al. Implementation of clinical guidelines via a computer charting system: effects on the care of febrile children less than three years of age. J Am Med Inform Assoc 2000;7:186-95.
113. Lee KL, Pryor DB, Harrell FE Jr, et al. Predicting outcome in coronary disease: statistical models versus expert clinicians. Am J Med 1986;80:553-60.
114. Hlatky MA, Califf RM, Harrell FE Jr, et al. Comparison of predictions based on observational data with the results of randomized controlled clinical trials of coronary artery bypass surgery. J Am Coll Cardiol 1988;11:237-45.
115. Vall Mayans M, Hall AJ, Inskip HM, et al. Do bedbugs transmit hepatitis B?. Lancet 1994;343:761-3.