Evidence-based health informatics: How do we know what we know?

Methods of Information in Medicine

Volumn 54, Issue 4, 2015, Pages 298-307

  Ammenwerth, E ^a  

^a UNIVERSITY FOR HEALTH SCIENCES   (Austria)

Author keywords

Evaluation studies; Evidence based medicine; Information science; Medical informatics; Meta analysis; Publication bias; Qualitative research; Review literature

Indexed keywords

EVIDENCE BASED MEDICINE; HISTORY; MASS COMMUNICATION; MEDICAL INFORMATICS; META ANALYSIS; MOTIVATION;

DIFFUSION OF INNOVATION; EVIDENCE-BASED MEDICINE; GOALS; HISTORY, 20TH CENTURY; HISTORY, 21ST CENTURY; MEDICAL INFORMATICS;

EID: 84938769604     PISSN: 00261270     EISSN: 2511705X     Source Type: Journal    
DOI: 10.3414/ME14-01-0119     Document Type: Article

References (95)

1. Jones SS, Rudin RS, Perry T, Shekelle PG. Health information technology: an updated systematic review with a focus on meaningful use. Ann Intern Med 2014; 160 (1): 48–54.
2. Institute of Medicine. Health IT and Patient Safety: Building Safer Systems for Better Care. Washington, D.C.: The National Academic Press; 2011.
3. Dowling AF, Jr. Do hospital staff interfere with computer system implementation? Health Care Manage Rev 1980; 5 (4): 23–32.
4. Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC, et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (6): 1506 –1512.
5. Magrabi F, Ong MS, Runciman W, Coiera E. Patient safety problems associated with heathcare information technology: an analysis of adverse events reported to the US Food and Drug Administration. AMIA Annu Symp Proc 2011; 2011: 853–857.
6. ECRI Institute. Deep Dive: Health Information Technology. 2012 [accessed Jan 14, 2015]; Available from: http://www.healthit.gov/facas/sites/ faca/files/STF_Deep_Dive_Health_Information_ Technology_2014-06-13.pdf
7. Goodman KW, Adams S, Berner ES, Embi PJ, Hsiung R, Hurdle J, et al. AMIA’s code of professional and ethical conduct. J Am Med Inform Assoc 2013; 20 (1): 141–143.
8. Haux R. Medical informatics: past, present, future. Int J Med Inform 2010; 79 (9): 599– 610.
9. Koppel R. Is healthcare information technology based on evidence? Yearb Med Inform 2013; 8 (1): 7–12.
10. Sackett D, Rosenberg W, Gray J, Haynes R, Richardson S. Evidence based medicine: what it is and what it isn’t. BMJ 1996; 312 (7023): 71–72.
11. Ammenwerth E, de Keizer N. A viewpoint on evidence-based health informatics, based on a pilot survey on evaluation studies in health care informatics. J Am Med Inform Assoc 2007; 14 (3): 368–371.
12. Ware GO, Park MK. Evaluation of search time for two computerized information retrieval systems at the University of Georgia. J Chem Doc 1972; 12 (4): 224 –227.
13. McFarlane AH, Norman GR. A medical care information system: evaluation of changing patterns of primary care. Med Care 1972; 10 (6): 481– 487.
14. van der Loo R. Overview of Published Assessment and Evaluation Studies. In: van Gennip EMSJ, Talmon JS, editors. Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995. pp 261–282.
15. Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care: Trends in evaluation research 1982 – 2002. Methods Inf Med 2005; 44: 44 –56.
16. Krobock JR. A taxonomy: hospital information systems evaluation methodologies. J Med Syst 1984; 8 (5): 419 – 429.
17. Gremy F. Hardware, software, peopleware, subjectivity. A philosophical promenade. Methods Inf Med 2005; 44 (3): 352–358.
18. Flagle C, Grémy F, Perry S, editors. Assessment of Medical Informatics Technology Joint Working Conference Montpellier du 22 au 26 octobre 1990. Rennes: Éditions ENSP; 1991.
19. Anderson JG, Aydin CE, Jay SJ, editors. Evaluating Health Care Information Systems – Methods and Applications. London, New Delhi: Sage Publications; 1994.
20. van Gennip E, Talmon J, editors. Assessment and evaluation of information technologies in medicine. Amsterdam: IOS Press; 1995.
21. Lorenzi NM, Riley RT. Managing Change: An Overview. J Am Med Inform Assoc 2000; 7 (2): 116 –124.
22. Anderson J, Aydin C, editors. Evaluating the Organizational Impact of Healthcare Information Systems. New York: Springer; 2005.
23. Friedman C, Wyatt JC. Evaluation Methods in Medical Informatics. 2nd ed. New York: Springer; 2006.
24. Brender J. Handbook of evaluation methods for health informatics. Burlington, MA: Elsevier Academic Press; 2006.
25. Rigby M, Ammenwerth E, Beuscart-Zephir M, Brender J, Hyppönen H, Melia S, et al. Evidence Based Health Informatics: 10 years of efforts to promote the principle. Yearb Med Inform 2013; 8 (1): 34– 46.
26. Haynes RB, Hayward RS, Jadad AR, Sebaldt RJ. Evidence based health informatics: an overview of the Health Information Research Unit at McMaster University. Leadersh Health Serv 1996; 5 (3): 41– 44.
27. Rigby M. Evaluation: 16 Powerful Reasons Why Not to Do It – And 6 Over-Riding Imperatives. In: Patel V, Rogers R, Haux R, editors. Proceedings of the 10th World Congress on Medical Informatics (Medinfo 2001). Amsterdam: IOS Press; 2001. pp 1198–1202.
28. Mantas J, Ammenwerth E, Demiris G, Hasman A, Haux R, Hersh W, et al. Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics. First Revision. Methods Inf Med 2010; 49 (2): 105 –120.
29. Seroussi B, Jaulent MC, Lehmann CU. Looking for the evidence: value of health informatics. Editorial. Yearb Med Inform 2013; 8 (1): 4 – 6.
30. UMIT, EFMI WG Eval. Health IT Evaluation Database. 2014 [accessed Jan 14, 2015]. Available from: http://evaldb.umit.at
31. Ammenwerth E, Gräber S, Herrmann G, Bürkle T, König J. Evaluation of Health Information Systems – Problems and Challenges. Int J Med Inform 2003; 71 (2–3): 125 –135.
32. Heathfield H, Buchan I. Current evaluations of information technology in health care are often inadequate. BMJ 1996; 313 (7063): 1008.
33. Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ 1998; 316: 1959 –1961.
34. Kaplan B, Duchon D. Combining qualitative and quantitative approaches in information systems research: a case study. MIS Quarterly 1988; 12 (4): 571–586.
35. Heathfield H, Peel V, Hudson P, Kay S, Mackay L, Marley T, et al. Evaluating Large Scale Health Information Systems: From Practice Towards Theory. In: Masys D, editor. AMIA Annual Fall Symposium. Philadelphia: Hanley & Belfus; 1997. pp 116 –120.
36. Sim J, Sharp K. A critical appraisal of the role of triangulation in nursing research. Int J Nurs Stud 1998; 35 (1–2): 23–31.
37. Barbour RS. Mixing qualitative methods: quality assurance or qualitative quagmire? Qual Health Res 1998; 8 (3): 352–361.
38. Grémy F, Degoulet P. Assessment of health information technology: which questions for which systems? Proposal for a taxonomy. Med Inform 1993; 18 (3): 185 –193.
39. Kaplan B. An Evaluation Model for Clinical Information Systems: Clinical Imaging Systems. In: Greenes R, Peterson H, Protti D, editors. Medinfo 95 – Proceedings of the 8th World Congress on Medical Informatics. Amsterdam: North Holland; 1995. p 1087.
40. AHRQ. Ageny for Healthcare Research and Quality: AHRQ Evaluation Toolkit. 2009 [accessed Jan 14, 2015]. Available from: http://healthit. ahrq.gov/health-it-tools-and-resources/health-itevaluation-toolkit-and-evaluation-measuresquick-reference
41. Nykanen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir MC, et al. Guideline for Good Evaluation Practice in Health Informatics (GEP-HI). Int J Med Inform 2011; 80: 815–827.
42. Malicki M, Marusic A, Consortium O. Is there a solution to publication bias? Researchers call for changes in dissemination of clinical research results. J Clin Epidemiol 2014; 67 (10): 1103 –1110.
43. Friedman C, Wyatt J. Publication bias in Medical Informatics. J Am Med Inform Assoc 2001; 8 (2): 189–191.
44. Wager E, Williams P, Consortium of Project Overcome failure to Publish Negative Findings. “Hardly worth the effort?” Medical journals’ policies and their editors’ and publishers’ views on trial registration and publication bias: quantitative and qualitative study. BMJ 2013; 347: f5248.
45. Eysenbach G. Tackling publication bias and selective reporting in health informatics research: register your eHealth trials in the International eHealth Studies Registry. J Med Internet Res 2004; 6 (3): e35.
46. Ammenwerth E, Schnell-Inderst P, Siebert U. Vision and challenges of Evidence-Based Health Informatics: a case study of a CPOE meta-analysis. Int J Med Inform 2010; 79 (4): e83–8.
47. Peute LW, Driest KF, Marcilly R, Bras Da Costa S, Beuscart-Zephir MC, Jaspers MW. A framework for reporting on human factor/usability studies of health information technologies. Stud Health Technol Inform 2013; 194: 54 – 60.
48. de Keizer NF, Ammenwerth E. The quality of evidence in health informatics: how did the quality of healthcare IT evaluation publications develop from 1982 to 2005? Int J Med Inform 2008; 77 (1): 41– 49.
49. Schulz KF, Altman DG, Moher D, Group C. CON SORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010; 340: c332.
50. Bossuyt PM, Reitsma J, Bruns D, Gatsonis C, Glasziou P, Irwig L, et al. Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative. Ann Int Med 2003; 138 (1): 40 – 44.
51. von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet 2007; 370 (9596): 1453 –1457.
52. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 2010; 8 (5): 336 –341.
53. EQUATOR. The Equator Network: Enhancing the Quality and Transparency of Health Research. 2014 [accessed Jan 14, 2015]. Available from:www. equator-network.org/
54. Talmon J, Ammenwerth A, Brender J, de Keizer N, Nykänen P, Rigby M. STARE-HI – Statement on Reporting of Evaluation Studies in Health Informatics. Int J Med Inform 2009; 78 (1): 1– 9.
55. Brender J, Talmon J, de Keizer N, Nykanen P, Rigby M, Ammenwerth E. STARE-HI – Statement on Reporting of Evaluation Studies in Health Informatics: explanation and elaboration. Appl Clin Inform 2013; 4 (3): 331–358.
56. Dixon BE, Zafar A, McGowan JJ. Development of a taxonomy for health information technology. Stud Health Technol Inform 2007; 129 (Pt 1): 616–620.
57. AHRQ. Agency for Healthcare Research and Qual ity: Health IT Costs and Benefit Database. 2009 [accessed Jan 14, 2015]. Available from: http:// healthit.ahrq.gov/health-it-tools-and-resources/ health-it-costs-and-benefits-database
58. ISO. ISO/TR 14639:2012: Health informatics – Capacity-based eHealth architecture roadmap; 2012.
59. Cravens GD, Dixon BE, Zafar A, McGowan JJ. A health information technology glossary for novices. AMIA Annu Symp Proc; 2008. p 917.
60. Higgins J, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. 2011 [accessed Jan 14, 2015]. Available from: http://www.cochranehandbook.org
61. Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U. The Effect of Electronic Prescribing on Medication Errors and Adverse Drug Events: A Systematic Review. J Am Med Inform Assoc 2008; 15 (5): 585– 600.
62. Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of clinical decisionsupport systems: a systematic review. Ann Intern Med 2012; 157 (1): 29 – 43.
63. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (10): 742–752.
64. Vervloet M, Linn AJ, van Weert JC, de Bakker DH, Bouvy ML, van Dijk L. The effectiveness of interventions using electronic reminders to improve adherence to chronic medication: a systematic review of the literature. J Am Med Inform Assoc 2012; 19 (5): 696 –704.
65. Cappuccio FP, Kerry SM, Forbes L, Donald A. Blood pressure control by home monitoring: meta-analysis of randomised trials. Bmj 2004; 329 (7458): 145.
66. Shea S, DuMouchel W, Bahamonde L. A metaanalysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc 1996; 3 (6): 399 – 409.
67. Walton R, Dovey S, Harvey E, Frreemantle N. Computer support for determining drug dose: systematic review and meta-analysis. BMJ 1999; 318: 984–990.
68. Mavridis D, Salanti G. How to assess publication bias: funnel plot, trim-and-fill method and selection models. Evid Based Ment Health 2014; 17 (1): 30.
69. Jones ML. Application of systematic review methods to qualitative research: practical issues. J Adv Nurs 2004; 48 (3): 271–278.
70. Sandelowski M, Barroso J, Voils CI. Using qualitative metasummary to synthesize qualitative and quantitative descriptive findings. Res Nurs Health 2007; 30 (1): 99–111.
71. Mantas J, Ammenwerth E, Demiris G, Hasman A, Haux R, Hersh W, et al. Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics. First Revision. Methods Inf Med 2010; 49 (2): 105–120.
72. Ammenwerth E, Craven C, Georgiou A, Mantas J. Health IT Evaluation in Health Informatics Curricula: International Overview and Recommendations. Workshop at Medical Informatics Europa (MIE2014), 1.9.2014, Istanbul 2014 [accessed Jan 14, 2015]. Available from: http://person.hst.aau. dk/ska/MIE2014/WorkshopsAndPanels/W03_ ID_472.pdf
73. Kucher N, Koo S, Quiroz R, Cooper JM, Paterno MD, Soukonnikov B, et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med 2005; 352 (10): 969–977.
74. Cresswell KM, Bates DW, Williams R, Morrison Z, Slee A, Coleman J, et al. Evaluation of mediumterm consequences of implementing commercial computerized physician order entry and clinical decision support prescribing systems in two ‘early adopter’ hospitals. J Am Med Inform Assoc 2014; 21 (e2): e194 –202.
75. Eslami S, Abu-Hanna A, de Keizer NF, de Jonge E. Errors associated with applying decision support by suggesting default doses for aminoglycosides. Drug Saf 2006; 29 (9): 803–809.
76. Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: findings from a mixed methods exploration. Int J Med Inform 2009; 78 (Suppl 1): S69–76.
77. Durieux P. Electronic medical alerts--so simple, so complex. N Engl J Med 2005; 352 (10): 1034 –1036.
78. Gross PA, Greenfield S, Cretin S, Ferguson J, Grimshaw J, Grol R, et al. Optimal methods for guideline implementation: conclusions from Leeds Castle meeting. Med Care 2001; 39 (8 Suppl 2): II85–92.
79. Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet 2003; 362 (9391): 1225 –1230.
80. Grol R, Grimshaw J. Evidence-based implementation of evidence-based medicine. Jt Comm J Qual Improv 1999; 25 (10): 503–513.
81. FDA. MAUDE – Manufacturer and User Facility Device Experience Database. 2014 [accessed Jan 14, 2015]. Available from: http://www.accessdata. fda.gov/scripts/cdrh/cfdocs/cfmaude/search.cfm
82. Magrabi F, Ong MS, Runciman W, Coiera E. Using FDA reports to inform a classification for health information technology safety problems. J Am Med Inform Assoc 2012; 19 (1): 45–53.
83. Meeks DW, Smith MW, Taylor L, Sittig DF, Scott JM, Singh H. An analysis of electronic health record-related patient safety concerns. J Am Med Inform Assoc 2014; 21 (6): 1053–1059.
84. Lovis C. Evidence-based Biomedical Informatics. The Long Way from Pioneer to Science. Yearb Med Inform 2013; 8 (1): 47–50.
85. Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information systemrelated errors. J Am Med Inform Assoc 2004; 11 (2): 104 –112.
86. Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of Unintended Consequences Related to Computerized Provider Order Entry. J Am Med Inform Assoc 2006; 13 (5): 547–556.
87. van der Sijs H, Kowlesar R, Aarts J, Berg M, Vulto A, van Gelder T. Unintended consequences of reducing QT-alert overload in a computerized physician order entry system. Eur J Clin Pharmacol 2009; 65 (9): 919–925.
88. Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform 2005; 74 (7–8): 519–526.
89. Low DK, Reed MA, Geiduschek JM, Martin LD. Striving for a zero-error patient surgical journey through adoption of aviation-style challenge and response flow checklists: a quality improvement project. Paediatr Anaesth 2013; 23 (7): 571–578.
90. Randell R. Medicine and aviation: a review of the comparison. Methods Inf Med 2003; 42 (4): 433–436.
91. Sullivan F. What is health informatics? J Health Serv Res Policy 2001; 6 (4): 251–254.
92. Uckert F, Ammenwerth E, Dujat C, Grant A, Haux R, Hein A, et al. Past and next 10 years of medical informatics. J Med Syst 2014; 38 (7): 74.
93. Hoyt R. Evidence-based health informatics: Replacing hype with science. 2013 [accessed Jan 14, 2015]. Available from: https://prezi.com/zsbc329– 25db/evidence-based-health-informatics
94. Godlee F. How guidelines can fail us. BMJ 2014; 349: g5448.
95. Cochrane A. Effectiveness and Efficiency: Random Reflections on Health Services. London: Nuffield Provincial Hospitals Trust; 1972.