Features of effective computerised clinical decision support systems: Meta-regression of 162 randomised trials

BMJ (Online)

Volumn 346, Issue 7899, 2013, Pages

  Roshanov, Pavel S ^a   Fernandes, Natasha ^b   Wilczynski, Jeff M ^c   Hemens, Brian J ^c   You, John J ^c,e   Handler, Steven M ^d   Nieuwlaat, Robby ^c,d   Souza, Nathan M ^c   Beyene, Joseph ^c,d   Van Spall, Harriette G C ^c,e   Garg, Amit X ^c,f   Haynes, R Brian ^c  

^a WESTERN UNIVERSITY   (Canada)

^b UNIVERSITY OF OTTAWA   (Canada)

^c MCMASTER UNIVERSITY   (Canada)

^d UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

^e MCMASTER UNIVERSITY   (Canada)

^f UNIVERSITY OF WESTERN ONTARIO   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ACCURACY; ARTICLE; CLINICAL PRACTICE; DECISION SUPPORT SYSTEM; HEALTH CARE; HUMAN; INTERMETHOD COMPARISON; MEDICAL DECISION MAKING; MEDICAL RECORD; META ANALYSIS; PATIENT CARE; PRIORITY JOURNAL; RANDOMIZED CONTROLLED TRIAL (TOPIC); SENSITIVITY ANALYSIS; STATISTICAL ANALYSIS; SYSTEMATIC REVIEW (TOPIC); TREATMENT OUTCOME; VALIDATION PROCESS;

DECISION SUPPORT SYSTEMS, CLINICAL; HUMANS; LOGISTIC MODELS; MULTIVARIATE ANALYSIS; OUTCOME AND PROCESS ASSESSMENT (HEALTH CARE); RANDOMIZED CONTROLLED TRIALS AS TOPIC; TECHNOLOGY ASSESSMENT, BIOMEDICAL; USER-COMPUTER INTERFACE;

EID: 84875164117     PISSN: None     EISSN: 17561833     Source Type: Journal    
DOI: 10.1136/bmj.f657     Document Type: Article

References (69)

1. Committee on Quality of Health Care in America Institute of Medicine. Crossing the quality chasm: a new health system for the 21st century. National Academies Press, 2001.
2. Committee on Quality of Health Care in America Institute of Medicine. Kohn L, Corrigan JM, Donaldson MS, eds. To err is human: building a safer health system. National Academies Press, 2000.
3. Blumenthal D, Tavenner M. The meaningful use regulation for electronic health records. N Engl J Med 2010; 363:501-4.
4. Hsiao C, Hing E. Use and characteristics of electronic health record systems among office-based physician practices: United States, 2001-2012. NCHS Data Brief 2012; 111. www.cdc.gov/nchs/data/databriefs/db111.htm
5. Kenealy T, Arroll B, Petrie KJ Patients and computers as reminders to screen for diabetes in family practice. Randomized-controlled trial. J Gen Intern Med 2005; 20:916-21. (Pubitemid 41726676)
6. Peterson KA, Radosevich DM, O'Connor PJ, Nyman JA, Prineas RJ, Smith S.A., et al. Improving diabetes care in practice: findings from the TRANSLATE trial. Diabetes Care 2008; 31:2238-43.
7. Cavalcanti AB, Silva E, Pereira A.J., Caldeira-Filho M, Almeida FP, Westphal GA, et al A randomized controlled trial comparing a computer-assisted insulin infusion protocol with a strict and a conventional protocol for glucose control in critically ill patients. J Crit Care 2009; 24:371-8.
8. Hemens BJ, Holbrook AM, Tonkin M, Mackay J.A., Weise-Kelly L, Navarro T, et al Computerized clinical decision support systems for drug prescribing and management: A decision-maker-researcher partnership systematic review. Implement Sci 2011; 6:89.
9. Roshanov PS, Misra S, Gerstein H.C., Garg AX, Sebaldt RJ, Mackay JA, et al Computerized clinical decision support systems for chronic disease management: a decision-maker-researcher partnership systematic review. Implement Sci 2011; 6:92.
10. Roshanov PS, You JJ, Dhaliwal J, Koff D., Mackay JA, Weise-Kelly L, et al Can computerized clinical decision support systems improve practitioners' diagnostic test ordering behavior? A decision-maker-researcher partnership systematic review. Implement Sci 2011; 6:88.
11. Nieuwlaat R, Connolly S, Mackay J.A., Weise-Kelly L, Navarro T, Wilczynski N.L., et al. Computerized clinical decision support systems for therapeutic drug monitoring and dosing: a decision-maker-researcher partnership systematic review. Implement Sci 2011; 6:90.
12. Sahota N, Lloyd R, Ramakrishna A., Mackay J, Prorok J, Weise-Kelly L, et al. Computerized clinical decision support systems for acute care management: a decision-maker-researcher partnership systematic review of effects on process of care and patient outcomes. Implement Sci 2011; 6:91.
13. Souza NM, Sebaldt RJ, Mackay J.A., Prorok J., Weise-Kelly L, Navarro T, et al Computerized clinical decision support systems for primary preventive care: A decision-maker-researcher partnership systematic review of effects on process of care and patient outcomes. Implement Sci 2011; 6:87.
14. Haynes RB, Wilczynski NL, the Computerized Clinical Decision Support System (CCDSS) Systematic Review Team. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: methods of a decision-maker-researcher partnership systematic review. Implement Sci 2010; 5:12.
15. Shiffman RN, Brandt CA, Liaw Y, Corb GJ A design model for computer-based guideline implementation based on information management services. J Am Med Inform Assoc 1999; 6:99-103. (Pubitemid 29122816)
16. Sim I, Gorman P, Greenes R.A., Haynes RB, Kaplan B, Lehmann H, et al Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc 2001; 8:527-34. (Pubitemid 33095492)
17. Solberg LI, Brekke ML, Fazio C.J., Fowles J., Jacobsen DN, Kottke TE, et al Lessons from experienced guideline implementers: attend to many factors and use multiple strategies. Jt Comm J Qual Improv 2000; 26:171-88.
18. Trivedi M, Kern J, Marcee A., Grannemann B, Kleiber B, Bettinger T., et al. Development and implementation of computerized clinical guidelines: barriers and solutions. Methods Inf Med 2002; 41:435-42. (Pubitemid 35453142)
19. Wetter T. Lessons learnt from bringing knowledge-based decision support into routine use. Artif Intell Med 2002; 24:195-203. (Pubitemid 34173631)
20. Balas EA, Weingarten S, Garb C.T., Blumenthal D., Boren SA, Brown GD Improving preventive care by prompting physicians. Arch Intern Med 2000; 160:301.
21. Kawamoto K, Houlihan CA, Balas E.A., Lobach DF Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330:765-72.
22. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano M, Devereaux P., Beyene J, et al Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293:1223-38. (Pubitemid 40327908)
23. Mollon B, Chong JJR, Holbrook AM, Sung M, Thabane L., Foster G. Features predicting the success of computerized decision support for prescribing: a systematic review of randomized controlled trials. BMC Med Inform Decis Mak 2009; 9:11.
24. Shojania KG, Jennings A, Mayhew A., Ramsay C, Eccles M, Grimshaw J. Effect of point-of-care computer reminders on physician behaviour: a systematic review. Can Med Assoc J 2010; 182:e216-25.
25. McKibbon KA, Lokker C, Handler S.M., Dolovich LR, Holbrook AM, O'Reilly D, et al The effectiveness of integrated health information technologies across the phases of medication management: a systematic review of randomized controlled trials. J Am Med Inform Assoc 2012; 19:22-30.
26. Sicotte C, Jaana M, Girouard D. Prioritizing the risk factors influencing the success of clinical information system projects: a Delphi study in Canada. Methods Inf Med 2008; 47:251-9. (Pubitemid 351723970)
27. Maiti T, Pradhan V. A comparative study of the bias corrected estimates in logistic regression. Stat Methods Med Res 2008; 17:621-34.
28. Heinze G, Schemper M. A solution to the problem of separation in logistic regression. Stat Med 2002; 21:2409-19.
29. Firth D. Bias reduction of maximum likelihood estimates. Biometrika 1993; 80:27.
30. Steyerberg EW, Harrell FE, Borsboom G.J., Eijkemans MJ, Vergouwe Y, Habbema JD Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 2001; 54:774-81. (Pubitemid 32722180)
31. Beyene J, Atenafu EG, Hamid J.S., To T., Sung L. Determining relative importance of variables in developing and validating predictive models. BMC Med Res Methodol 2009; 9:64.
32. White IR, Royston P, Wood AM Multiple imputation using chained equations: Issues and guidance for practice. Stat Med 2011; 30:377-99.
33. Stata Statistical Software: release 11. StataCorp; 2009.
34. Kesselheim AS, Cresswell K, Phansalkar S., Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce "alert fatigue" while still minimizing the risk of litigation. Health Aff (Millwood) 2011; 30:2310-7.
35. Weingart SN, Toth M, Sands D.Z., Aronson MD, Davis RB, Phillips RS Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003; 163:2625-31. (Pubitemid 37464421)
36. Isaac T, Weissman JS, Davis R.B., Massagli M., Cyrulik A, Sands DZ, et al Overrides of medication alerts in ambulatory care. Arch Intern Med 2009; 169:305-11.
37. Van Der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13:138-47. (Pubitemid 43332904)
38. Scott GPT, Shah P, Wyatt J.C., Makubate B., Cross FW Making electronic prescribing alerts more effective: scenario-based experimental study in junior doctors. J Am Med Inform Assoc 2011; 18:789-98.
39. Strom BL, Schinnar R, Aberra F., Bilker W, Hennessy S, Leonard C.E., et al. Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med 2010; 170:1578-83.
40. Phansalkar S, Edworthy J, Hellier E., Seger DL, Schedlbauer A, Avery AJ, et al A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc 2010; 17:493-501.
41. Ash JS, McCormack JL, Sittig D.F., Wright A., McMullen C, Bates DW Standard practices for computerized clinical decision support in community hospitals: a national survey. J Am Med Inform Assoc 2012; 19:980-7.
42. Peduzzi P, Concato J, Kemper E., Holford TR, Feinstein AR A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996; 49:1373-9. (Pubitemid 27028925)
43. Steyerberg EW, Schemper M, Harrell FE Logistic regression modeling and the number of events per variable: selection bias dominates. J Clin Epidemiol 2011; 64:1464-5.
44. Babyak MA. What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models. Psychosom Med 2004; 66:411-21. (Pubitemid 38668414)
45. Courvoisier DS, Combescure C, Agoritsas T., Gayet-Ageron A, Perneger TV Performance of logistic regression modeling: beyond the number ofevents per variable, the role of data structure. J Clin Epidemiol 2011; 64:993-1000.
46. Thompson SG, Higgins JPT. How should meta-regression analyses be undertaken and interpreted? Stat Med 2002; 21:1559-73. (Pubitemid 34746063)
47. Negassa A, Hanley JA The effect of omitted covariates on confidence interval and study power in binary outcome analysis: a simulation study. Contemp Clin Trials 2007; 28:242-8. (Pubitemid 46330167)
48. Liu JLY, Wyatt JC The case for randomized controlled trials to assess the impact of clinical information systems. J Am Med Inform Assoc 2011; 18:173-80.
49. Tierney W, Overhage J, McDonald C. A plea for controlled trials in medical informatics. J Am Med Inform Assoc 1994; 1:353-5.
50. Stead WW, Haynes RB, Fuller S, Friedman C.P., Travis LE, Beck JR, et al Designing medical informatics research and library - resource projects to increase what is learned. J Am Med Inform Assoc 1994; 1:28-34.
51. Flottorp S, Havelsrud K, Oxman AD Process evaluation of a cluster randomized trial of tailored interventions to implement guidelines in primary care - why is it so hard to change practice? Fam Pract 2003; 20:333-9. (Pubitemid 36722472)
52. Wright A, Sittig DFF. A four-phase model of the evolution of clinical decision support architectures. Int J Med Inf 2008; 77:641-9.
53. McDonald CJJ. Use of a computer to detect and respond to clinical events: its effect on clinician behavior. Ann Intern Med 1976; 84:162-7.
54. McDonald CJ, Tierney WM The Medical Gopher - a microcomputer system to help find, organize and decide about patient data. West J Med 1986; 145:823-9. (Pubitemid 17226203)
55. Tierney WM, Overhage JM, Murray M.D., Harris LE, Zhou XH, Eckert GJ, et al Effects of computerized guidelines for managing heart disease in primary care. J Gen Intern Med 2003; 18:967-6.
56. Tierney WM, Overhage JM, Murray M.D., Harris LE, Zhou XH, Eckert GJ, et al Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial. Health Serv Res 2005; 40:477-97. (Pubitemid 40470704)
57. Mamlin BW, Overhage JM, Tierney W, Dexter P., McDonald CJ Clinical decision support within the regenstrief medical record system. In: Berner ES, ed. Clinical decision support systems: theory and practice. 2nd ed. Springer, 2007:190-214.
58. Tierney WM, Miller ME, Overhage J.M., McDonald CJ Physician inpatient order writing on microcomputer workstations. Effects on resource utilization. JAMA 1993; 269:379-83. (Pubitemid 23018077)
59. Overhage JM, Tierney WM, Zhou X.H., McDonald CJ A randomized trial of "corollary orders" to prevent errors of omission. J Am Med Inform Assoc 1997; 4:364-75. (Pubitemid 127433631)
60. Tierney WM, McDonald CJ, Hui S.L., Martin DK Computer predictions of abnormal test results: effects on outpatient testing. JAMA 1988; 259:1194-8. (Pubitemid 18058831)
61. Tierney WM, Miller ME, McDonald CJ The effect on test ordering of informing physicians of the charges for outpatient diagnostic tests. N Engl J Med 1990; 322:1499-504. (Pubitemid 20179864)
62. Holbrook A, Keshavjee K, Lee H., Bernstein B, Chan D, Thabane L., et al. Individualized electronic decision support and reminders can improve diabetes care in the community. AMIA Annu Symp Proc 2005:982.
63. Holbrook A, Thabane L, Keshavjee K., Dolovich L, Bernstein B, Chan D., et al. Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial. Can Med Assoc J 2009; 181:37-44.
64. Lester WT, Grant R, Barnett G.O., Chueh H. Facilitated lipid management using interactive e-mail: preliminary results of a randomized controlled trial. Stud Health Technol Inform 2004; 107:232-6.
65. Lester WT, Grant RW, Barnett G.O., Chueh HC Randomized controlled trial of an informatics-based intervention to increase statin prescription for secondary prevention of coronary disease. J Gen Intern Med 2006; 21:22-9.
66. Quinn CC, Clough SS, Minor J.M., Lender D., Okafor MC, Gruber-Baldini A. WellDoc mobile diabetes management randomized controlled trial: change in clinical and behavioral outcomes and patient and physician satisfaction. Diabetes Technol Ther 2008; 10:160-8. (Pubitemid 351677598)
67. Lee N-J, Chen E.S., Currie LM, Donovan M, Hall EK, Jia H, et al The effect of a mobile clinical decision support system on the diagnosis of obesity and overweight in acute and primary care encounters. ANS Adv Nurs Sci 2009; 32:211-21.
68. Osheroff JA, Teich JM, Middleton B, Steen E.B., Wright A., Detmer DE A roadmap for national action on clinical decision support. J Am Med Inform Assoc 2007; 14:141-5. (Pubitemid 46275496)
69. Guidance Documents (Medical Devices and Radiation-Emitting Products). Draft Gguidance for industry and Food and Drug Administration staff - mobile medical applications. www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/ GuidanceDocuments/ucm263280.htm.