Evaluation of health information systems - Problems and challenges

International Journal of Medical Informatics

Volumn 71, Issue 2-3, 2003, Pages 125-135

  Ammenwerth, Elske ^a   Gräber, Stefan ^b   Herrmann, Gabriele ^c   Bürkle, Thomas ^d   König, Jochem ^b  

^a University for Medical Informatics Technology Tyrol   (Austria)

^b SAARLAND UNIVERSITY   (Germany)

^c UNIVERSITY OF LEIPZIG   (Germany)

^d UNIVERSITY OF MÜNSTER   (Germany)

Author keywords

Evaluation studies; Health care; Information systems; Information technology; Technology assessment

Indexed keywords

DECISION MAKING; INFORMATION TECHNOLOGY;

HEALTH INFORMATION SYSTEMS;

HEALTH;

CONFERENCE PAPER; EVALUATION; HEALTH CARE; HEALTH CARE QUALITY; MEDICAL INFORMATICS; MEDICAL INFORMATION SYSTEM; MEDICAL TECHNOLOGY; PRIORITY JOURNAL;

EID: 0141831925     PISSN: 13865056     EISSN: None     Source Type: Journal    
DOI: 10.1016/S1386-5056(03)00131-X     Document Type: Conference Paper

References (83)

1. Bates D.W., Cohen M., Leape L.L., Overhage J.M., Shabot M.M., Sheridan T. Reducing the frequency of errors in medicine using information technology. JAMIA. 8(4):2001;299-308.
2. HIMSS, The 11th Annual HIMSS Leadership Survey Sponsored by IBM: Trends in Healthcare Information and Technology - Final Report, Healthcare Information and Management Systems Society, http://www2.himss.org/survey/2000/survey2000.html (last access: January 2003).
3. M. Rigby, Evaluation: 16 powerful reasons why not to do it - and 6 over-riding imperatives, in: V. Patel, R. Rogers, R. Haux (Eds.), Proceedings of the 10th World Congress on Medical Informatics (MedInfo 2001), Studies in Health Technology and Informatics, vol. 84, IOS Press, Amsterdam, 2001, pp. 1198-1202.
4. B. Kaplan, N. Shaw, People, organizational and social issues: evaluation as an exemplar, in: R. Haux, C. Kulikowski (Eds.), Yearbook of Medical Informatics 2002, Schattauer, Stuttgart, 2002, pp. 91-102.
5. L. Heinrich, Informationsmanagement, Oldenbourg, München Wien, 1999.
6. Miller R., Gardner R. Recommendations for responsible monitoring and regulation of clinical software systems. JAMIA. 4(6):1997;442-457.
7. Winter A., Ammenwerth E., Bott O., Brigl B., Buchauer A., Gräber S., Grant A., Häber A., Hasselbring W., Haux R., Heinrich A., Janssen H., Kock I., Penger O.-S., Prokosch H.-U., Terstappen A., Winter A. Strategic information management plans: the basis for systematic information management in hospitals. Int. J. Med. Inf. 64(2-3):2001;99-109.
8. VATAM, VATAM guidelines, Validation of Health Telematics Applications (VATAM), http://www-vatam.unimaas.nl (last access: July 2003).
9. C. Friedman, J.C. Wyatt, Evaluation Methods in Medical Informatics, Springer, New York, 1997.
10. Holle R., Zahlmann G. Evaluation of telemedical services. IEEE Trans. Inf. Technol. Biomed. 3(2):1999;84-91.
11. Brender J. Trends in assessment of IT-based solutions in healthcare and recommendations for the future. Int. J. Med. Inf. 52(1-3):1998;217-227.
12. Chocholik J., Bouchard S., Tan J., Ostrow D. The determination of relevant goals and criteria used to select an automated patient care information system: a Delphi approach. JAMIA. 6(3):1999;219-233.
13. L. Einbinder, J. Remz, D. Cochran, Mapping clinical scenarios to functional requirements: a tool for evaluating clinical information systems, in: J. Cimino (Ed.), Proceedings of the AMIA Annual Fall Symposium, Hanley & Belfus, Philadelphia, 1996, pp. 747-751.
14. R. Stiefel, E. Rizkalla, The elements of a complete product evaluation, Biomed. Instrum. Technol. November-December (1995) 482-488.
15. M. Beuscart-Zéphir, P. Sockeel, B. Bossard, R. Beuscart, Activity modelling for assessing the usability of telematics applications in healthcare, in: B. Cesnik, A. McCray, J. Scherrer (Eds.), Proceedings of the 9th World Congress on Medical Informatics (MedInfo 1998), Studies in Health Technology and Informatics, vol. 52, IOS Press, Amsterdam, 1998, pp. 832-836.
16. Gräber S. Application of clinical workstations: functionality and usability. Clin. Perform. Qual. Health. 5(2):1997;71-75.
17. Kushniruk A.W., Patel C., Patel V.L., Cimino J.J. 'Televaluation' of clinical information systems: an integrative approach to assessing Web-based systems. Int. J. Med. Inf. 61(1):2001;45-70.
18. G. Braccini, F. Fabbrini, M. Fusani, Software quality assessment for health care systems, in: C. Pappas (Ed.), Proceedings of the Medical Informatics Europe 1997, Studies in Health Technology and Informatics, vol. 43, IOS Press, Amsterdam, 1997, pp. 746-750.
19. J. Anderson, Increasing the acceptance of clinical information systems, MD Comput. January-February (1999) 62-65.
20. May C., Gask L., Atkonson T., Ellis N., Mair F., Esmail A. Resisting and promoting new technologies in clinical practice: the case of telepsychiatry. Soc. Sci. Med. 52:2001;1889-1901.
21. Goodhue D. Understanding user evaluations of information systems. Manage. Sci. 41(12):1995;1827-1844.
22. D. Sittig, G. Kuperman, J. Fiskio, Evaluating physician satisfaction regarding user interactions with an electronic medical record system, in: Proceedings of the Annual Symposium on Computer Application in Medical Care, Hanley & Belfus, Philadelphia, 1999, pp. 400-404.
23. Ammenwerth E., Eichstädter R., Haux R., Pohl U., Rebel S., Ziegler S. A randomized evaluation of a computer-based nursing documentation system. Methods Inf. Med. 40(2):2001;61-68.
24. Bürkle T., Kuch R., Prokosch H., Dudeck J. Stepwise evaluation of information systems in an university hospital. Methods Inf. Med. 38(1):1999;9-15.
25. Herbst K., Littlejohns P., Rawlinson J., Collinson M., Wyatt J. Evaluating computerized health information systems: hardware, software and human ware: experiences from the Northern Province, South Africa. J. Public Health Med. 21(3):1999;305-310.
26. Shea S., Sideli R., DuMouchel W., Pulver G., Arons R., Clayton P. Computer-generated informational messages directed to physicians: effect on length of hospital stay. JAMIA. 2(1):1995;58-64.
27. Watkins J., Weatherburn G., Bryan S. The impact of a picture archiving and communication system (PACS) upon an intensive care unit. Eur. J. Radiol. 34(1):2000;3-8.
28. R. Currell, C. Urquhart, P. Wainwright, R. Lewis, Telemedicine versus face to face patient care: effects on professional practice and health care outcomes (Cochrane review), in: The Cochrane Library, Issue 1, Update Software, Oxford, 2000.
29. Gray J.E., Safran C., Davis R.B., Pompilio-Weitzner G., Stewart J.E., Zaccagnini L., Pursley D. Baby CareLink: using the internet and telemedicine to improve care for high-risk infants. Pediatrics. 106(6):2000;1318-1324.
30. McCowan C., Neville R.G., Ricketts I.W., Warner F.C., Hoskins G., Thomas G.E. Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma. Med. Inf. Internet. Med. 26(3):2001;191-201.
31. Dexter P.R., Perkins S., Overhage J.M., Maharry K., Kohler R.B., McDonald C.J. A computerized reminder system to increase the use of preventive care for hospitalized patients. N. Engl. J. Med. 345(13):2001;965-970.
32. Mair F., Whitten P. Systematic review of studies of patient satisfaction with telemedicine. BMJ. 320(7248):2000;1517-1520.
33. Nahm R., Poston I. Measurement of the effects of an integrated, point-of-care computer system on quality of nursing documentation and patient satisfaction. Comput. Nurs. 18(5):2000;220-229.
34. J. Enning, A. Bakker, Analysis of costs of information systems, in: E. van Gennip, J. Talmon (Eds.), Assessment and Evaluation Technologies in Medicine, Studies in Health Technology and Informatics, vol. 17, IOS Press, Amsterdam, 1995, pp. 87-97.
35. van Gennip E., Bakker A. Assessment of effects and costs of information systems. Int. J. Biomed. Comput. 39:1995;67-72.
36. Bryan S., Weatherburn G., Buxton M., Watkins J., Keen J., Muris N. Evaluation of a hospital picture archiving and communication system. J. Health Serv. Res. Policy. 4(4):1999;204-209.
37. Lock C. What values do computers provide to NHS hospitals? BMJ. 312:1996;1407-1410.
38. Mair F.S., Haycox A., May C., Williams T. A review of telemedicine cost-effectiveness studies. J. Telemed. Telecare. 6(Suppl. 1):2000;S38-S40.
39. Bryan S., Buxton M., Brenna E. Estimating the impact of a diffuse technology on the running costs of a hospital. A case study of a picture archiving and communication system. Int. J. Technol. Assess. Health Care. 16(3):2000;787-798.
40. R. van der Loo, Overview of published assessment and evaluation studies, in: E.M.S.J. van Gennip, J.S. Talmon (Eds.), Assessment and Evaluation of Information Technologies, Studies in Health Technology and Informatics, vol. 17, IOS Press, Amsterdam, 1995, pp. 261-282.
41. H. Heathfield, V. Peel, P. Hudson, S. Kay, L. Mackay, T. Marley, L. Nicholson, R. Roberts, J. Williams, Evaluating large scale health information systems: from practice towards theory, in: D. Masys (Ed.) Proceedings of the AMIA Annual Fall Symposium, Hanley & Belfus, Philadelphia, 1997, pp. 116-120.
42. T. Bürkle, M. Schmitz, H.U. Prokosch, J. Dudeck, A systematic approach for evaluation of nursing work in an university hospital, in: P. Barahona, M. Veloso, J. Bryant (Eds.), Medical Informatics Europe 1994, European Federation for Medical Informatics (EFMI), Lissabon, 1995, pp. 1321-1325.
43. Henderson R., Deane F. User expectations and perceptions of a patient management information system. Comput. Nurs. 14(3):1996;188-193.
44. Heathfield H., Pitty D., Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ. 316:1998;1959-1961.
45. Tierney W., Overhage J., McDonald C. A plea for controlled trials in medical informatics. JAMIA. 1(4):1994;353-355.
46. ICH-GCP, Good Clinical Practice (CPMP/ICH/135/95), International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use, Geneva, 1996 ( http://www.ifpma.org/ichl.html ).
47. Schäfer H., Berger J., Biebler K.-E., Feldmann U., Greiser E., Jöckel K.-H., Michaelis J., Neiss A., Rapse H.H., Robra B.-P., Schumacher M., Trampisch J.-J., Victor N., Windeler J. Empfehlungen für die Erstellung von Studienprotokollen (Studienplänen) für klinische Studien. Informatik Biometrie und Epidemiologie in Medizin und Biologie. 30(3):1999;141-154.
48. Moehr J.R. Evaluation: salvation or nemesis of medical informatics? Comput. Biol. Med. 32(3):2002;113-125.
49. Tierney W., McDonald C. Testing informatics innovations: the value of negative trials. JAMIA. 3(5):1996;358-359.
50. Berg M. Patient care information systems and healthcare work: a sociotechnical approach. Int. J. Med. Inf. 55:1999;87-101.
51. Palvia S., Sharma R., Conrath D. A sociotechnical framework for quality assessment of computer information systems. Ind. Manage. Data Syst. 101(5):2001;237-251.
52. Butler M., Bender A. Intensive care unit bedside documentation systems - realizing cost savings and quality improvements. Comput. Nurs. 17(1):1999;32-38.
53. C. Mahler, E. Ammenwerth, A. Wagner, A. Tautz, T. Happek, B. Hoppe, R. Eichstädter, Effects of a computer-based nursing documentation system on the quality of nursing documentation, J. Med. Syst., 2003, in press.
54. Ammenwerth E., Mansmann U., Iller C., Eichstädter R. Factors affecting and affected by user acceptance of computer-based nursing documentation: results of a two-year study. JAMIA. 10(1):2003;69-84.
55. Lorenzi N.M., Riley R.T. Managing change: an overview. JAMIA. 7(2):2000;116-124.
56. D.E. Forsythe, B.G. Buchanan, Broadening our approach to evaluating medical information systems, in: P. Clayton (Ed.), Proceedings of the 15th Annual Symposium on Computer Applications in Medical Care, McGraw-Hill, New York, 1992, pp. 8-12.
57. Wyatt J. Clinical data systems. Part 3. Development and evaluation. Lancet. 344:1994;1682-1688.
58. E. Ammenwerth, U. Mansmann, C. Mahler, M. Kandert, R. Eichstädter, Are quantitative methods sufficient to show why wards react differently to computer-based nursing documentation? in: G. Surjan, R. Engelbrecht, P. McNair (Eds.), Proceedings of the 17th International Congress of the European Federation for Medical Informatics (Medical Informatics Europe 2002 - Health Data in the Information Society), 25-29 August 2002, Budapest, Studies in Health Technology and Informatics, vol. 90, IOS Press, Amsterdam, 2002, pp. 377-381.
59. Heathfield H., Hudson P., Kay S., Mackay L., Marley T., Nicholson L., Peel V., Roberts R., Williams J. Issues in the multi-disciplinary assessment of healthcare information systems. Ass. Healthcare Inf. Technol. People. 12(3):1999;253-275.
60. T. Jorgensen, Measuring effects, in: E.M.S.J. van Gennip, J.L. Talmon (Eds.), Assessment and Evaluation of Information Technologies, Studies in Health Technology and Informatics, vol. 17, IOS Press, Amsterdam, 1995, pp. 99-109.
61. J. Frechtling, User-Friendly Handbook for Mixed Method Evaluation, http://www.ehr.nsf.gov/ehr/rec/pubs/nsf97-153/start.htm (last access: July 2003).
62. Dewan N., Lorenzi N. Behavioral health information systems: evaluating readiness and user acceptance. MD Comput. 17(4):2000;50-52.
63. E. Ammenwerth, R. Eichstädter, T. Happek, R. Haux, B. Hoppe, M. Kandert, A. Kutscha, G. Luther, C. Mahler, U. Mansmann, U. Pohl, Evaluation of computer-based documentation on four wards - final report (in German), Department of Medical Informatics, Heidelberg, 2001.
64. N. Denzin, Strategies of multiple triangulation, in: N. Denzing (Ed.), The Research Act, McGraw-Hill, New York, 1970, pp. 297-331.
65. Grémy F., Fessler J., Bonnin M. Information systems evaluation and subjectivity. Int. J. Med. Inf. 56:1999;13-23.
66. J. Wyatt, D. Spiegelhalter, Field trials of medical decision-aids: potential problems and solutions, in: P. Clayton (Ed.), Proceedings of the 15th Annual Symposium on Computer Applications in Medical Care, McGraw-Hill, New York, 1992, pp. 3-7.
67. Talmon J., Hasmann A. Medical informatics as a discipline at the beginning of the 21st century. Methods Inf. Med. 41:2002;4-7.
68. Shahar Y. Medical informatics: between science and engineering, between academia and industry. Methods Inf. Med. 41:2002;8-11.
69. Grémy F., Degoulet P. Assessment of health information technology: which questions for which systems? Proposal for a taxonomy. Med. Inf. (Lond.). 18(3):1993;185-193.
70. Johnston M., Langton K., Haynes R., Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome - a critical appraisal of research. Ann. Intern. Med. 120:1994;135-142.
71. Kaplan B. Evaluating informatics applications - clinical decision support systems literature review. Int. J. Med. Inf. 64:2001;15-37.
72. Balas E.A., Austin S.M., Mitchell J.A., Ewigman B.G., Bopp K.D., Brown G.D. The clinical value of computerized information services. A review of 98 randomized clinical trials. Arch. Fam. Med. 5:1996;27-278.
73. Walton R., Dovey S., Harvey E., Frreemantle N. Computer support for determining drug dose: systematic review and meta-analysis. BMJ. 318:1999;984-990.
74. Chan Y. IT value: the great divide between qualitative and quantitative and individual and organizational measures. J. Manage. Inf. Syst. 16(4):2000;225-261.
75. N. Doherty, L. Mcaulay, The Relationship Between the ex ante and ex post information systems evaluation: reflections from the literature and the practice of evaluating e-commerce investments, in: A. Brown, D. Remenyi (Eds.), Proceedings of the Ninth European Conference on Information Technology Evaluation (ECITE 2002), Université Paris-Dauphine, 15-16 July 2002, MCIL, Reading, 2002, pp. 129-136.
76. K. Grundén, An evaluation model for CSCW systems, in: A. Brown, D. Remenyi (Eds.), Proceedings of the Ninth European Conference on Information Technology Evaluation (ECITE 2002), Université Paris-Dauphine, 15-16 July 2002, MCIL, Reading, 2002, pp. 181-187.
77. Shaw N. 'CHEATS': a generic information communication technology (ICT) evaluation framework. Comput. Biol. Med. 32:2002;200-209.
78. Grant A., Plante I., Leblanc F. The TEAM methodology for the evaluation of information systems in biomedicine. Comput. Biol. Med. 32(3):2002;195-207.
79. C. Ohmann, G. Belenky. Leitfaden zur Evaluierung von Wissensbasen des MEDWIS-Arbeitskreises "Evaluation", http://www.uni-duesseldorf.de/www/medfak/theochir/ak_eval/lf_main.htm (last access: July 2003).
80. F. Eurlings, A. van Asten, H. Cozijn, K. Klaassen, R. Stokman, R. van Valkenburg, E. van Gennip, Effects of a nursing information system in 5 Dutch hospitals, in: U. Gerdin, M. Tallberg, P. Wainwright (Eds.), Nursing Informatics - The Impact of Nursing Knowledge on Health Care Informatics, Studies in Health Technology and Informatics, vol. 46, IOS Press, Amsterdam, 1997, pp. 50-55.
81. S. Sapirie, Assessing health information systems, in: T. Lippeveld, R. Sauerborn, C. Bodard (Eds.), Design and Implementation of Health Information Systems, World Health Organization, Geneva, 2000.
82. M. Hebert, Telehealth success: evaluation framework development, in: V. Patel, R. Rogers, R. Haux (Eds.), Proceedings of the 10th World Congress on Medical Informatics (MedInfo 2001), Studies in Health Technology and Informatics, vol. 84, IOS Press, Amsterdam, 2001, pp. 1145-1149.
83. Kaplan B., Brennan P., Dowling A., Friedman C., Peel V. Toward an informatics research agenda: key people and organizational issues. JAMIA. 8(3):2001;235-241.