Trends in assessment of IT-based solutions in healthcare and recommendations for the future

International Journal of Medical Informatics

Volumn 52, Issue 1-3, 1998, Pages 217-227

  Brender, Jytte ^a  

^a AALBORG UNIVERSITY   (Denmark)

Author keywords

Computer system; Constructive assessment; Evaluation; Formative assessment; Human aspects; IT system; Lifecycle; Organisational issues; Review; System development; Technology assessment

Indexed keywords

COMPUTER AIDED DIAGNOSIS; HEALTH CARE; INFORMATION TECHNOLOGY;

MEDICAL INFORMATICS;

MEDICAL COMPUTING;

CONFERENCE PAPER; HEALTH CARE; INFORMATION SCIENCE; LIFE CYCLE; PRIORITY JOURNAL; SYSTEM ANALYSIS;

COMPUTER SYSTEMS; CONGRESSES; EUROPE; FORECASTING; HUMANS; MEDICAL INFORMATICS; TECHNOLOGY ASSESSMENT, BIOMEDICAL;

EID: 0032191921     PISSN: 13865056     EISSN: None     Source Type: Journal    
DOI: 10.1016/S1386-5056(98)00140-3     Document Type: Conference Paper

References (74)

1. V. Laopodis, TAIS: Technology Assessment Information System. In: K. Duncan, K. Krueger (Eds.), Proceedings of the Thirteenth World Computer Congress 94, vol. 3, Elsevier, Amsterdam, 1994, pp. 235-240.
2. Foote S.B. Assessing medical technology assessment: past, present and future. Milbank Q. 65(1):1987;59-80.
3. C. Flagle, F. Grémy, S. Perry (Eds.), Assessment of Medical Informatics Technology, Joint Working Conference, Montpellier du 22 au 26 Octobre 90, Éditions ENSP, Rennes, 1991.
4. R.I. Benjamin, E. Levison, A Framework for Managing IT - Enables Change, Sloan Management Review, Summer 1993, pp. 23-33.
5. C. Nøhr, Computer-Assisted Decision-Support-Improved Health? A Prospective Technology Assessment. PhD Dissertation, University of Aalborg, Department of Development and Planning, 1991.
6. E.M.J.S. van Gennip, J. Talmon, (Eds), Assessment and Evaluation of Information Technologies, Studies in Health Technology and Informatics, vol. 17, IOS Press, Amsterdam, 1995.
7. J. Brender, J. Talmon, P. Nykänen, P. McNair, M. Demeester, R. Beuscart, On the Evaluation of System Integration, in: E.M.S.J. van Gennip, J. Talmon (Eds.), Assessment and Evaluation of Information Technologies, Studies in Health Technology and Informatics, vol. 17, IOS Press, Amsterdam, 1995, pp. 189-208.
8. J.H.T.H. Andriessen, I.E. van der Horst-Bruinsma, B.M. ter Haar Romeny, Methodology of PACS Effectiveness Evaluation as part of a Technology Assessment, The Dutch PACS Project Extrapolated, in: R.H. Schenider, S.J. Dwyer III, R.G. Jost (Eds.), Medical Imaging III: PACS System Design and Evaluation, Proceedings SPIE - The International Society for optical Engineering. vol. 1093, 1989, pp. 585-588.
9. J.G. Anderson, C.E. Aydin, S.J. Jay (Eds.), Evaluating Health Care Information Systems, Methods and Applications, SAGE, Thousand Oaks, 1994.
10. J. Brender, Methodology for Asssessment of Medical IT-based Solutions in an Organisational Context, Studies in Health Technology and Informatics, vol. 42, IOS Press, Amsterdam, 1997.
11. J.M.C. Bastien, D.L. Scapin, Ergonomic Criteria for the Evaluation of Human-Computer Interfaces, Technical Report, vol. 156, INRIA, 1993.
12. M.C. Beuscart-Zephir, J. Brender, R. Beuscart, I. Ménager-Depriester, Cognitive evaluation: how to assess the usability of information technology in healthcare, Comput. Methods Programs. Biomed. 54 (1-2) (1997) 19-28.
13. Brannigan V. Acceptance testing - the critical problem in software acquisition. IEEE Trans. Biomed. Eng. 32(4):1985;295-299.
14. Müller J., Kjaer-Rasmussen J., Nohr C., Proactive Technology Assessment as a Tool in HIS Development, in: B. Barber, D. Cao, D. Qin, G. Wagner (Eds.), MEDINFO 89, Proceedings of the Sixth Conference on Medical Informatics, vol. 1, Elsevier, Amsterdam, 1989, pp. 308-311.
15. Nykänen P., Chowdhury S., Wigertz O. Evaluation of decision support systems in medicine. Comput. Methods Programs Biomed. 34:1991;229-238.
16. R. O'Moore, K. Clarke, R. Smeets, J. Brender, P. Nykänen, P. McNair, J. Grimson, B. Barber, Items of Relevance for Evaluation of Knowledge-Based Systems and Influence from Domain Characteristics; Technical Report of the KAVAS (A1021) AIM Project, EM-1.1, 1990 (public).
17. Gupta U.G. Validation and verification of knowledge-based systems: a survey. J. Appl. Intell. 3:1993;343-363.
18. J. Brender, P. McNair, H. Raun, J. Nolan, S. Vingtoft, Meta-knowledge Concepts, Technical report of the KAVAS (A1021) AIM Project, META-1.1, 2nd ed., 1990 (public, available from the author).
19. O'Keefe R.M., O'Leary D.E. Expert system verification and validation: a survey and tutorial. Artif. Intell. Rev. 7:1993;3-42.
20. D.E. O'Leary, Verifying and Validating Expert Systems: A Survey, in: P.R. Watkins, L.B. Eliot (Eds.), Expert Systems in Business and Finance: Issues and Applications, Wiley, New York, 1993, pp. 181-208.
21. J.A. Cannatacci, Liability and Responsibility for Expert Systems, TERESA, vol. 54, TANO, Oslo, 1988.
22. Gemignani M.C. Some legal aspects of expert systems. Expert Syst. Appl. 2:1991;269-283.
23. C. Laske, Legal Issues in Medical Informatics: A Bird's Eye View, in: B. Barber, A. Treacher, K. Louwerse (Eds.), Towards Security in Medical Telematics, Legal and Technical Aspects, Studies in Health Technology and Informatics, vol. 27, IOS Press, Amsterdam, 1996, pp. 53-78.
24. C. Laske, Healthcare telematics, who is liable? Comput. Methods Programs Biomed. 54 (1-2) (1997) 1-6.
25. J. Talmon, J. Fox (Eds.), Knowledge-based Systems in Medicine: Methods, Applications and Evaluation, Proceedings of the Workshop 'System Engineering in Medicine', Maastricht, 1989, Lecture Notes in Medical Informatics, vol. 47, Springer, Berlin, 1991.
26. Wyatt J., Spiegelhalter D. Med. Inform. 15(3):1990;183-184.
27. P. Nykänen (Ed.), Issues in Evaluation of Computer-based Support to Clinical Decision Making, vol. 127, Oslo University Research Reports, Oslo, 1990.
28. J. Wyatt, The Evaluation of Clinical Decision Support Systems: A Discussion of the Methodology Used in the ACORN Project, in: J. Fox, M. Fieschi, R. Engelbrecht (Eds.), European Conference on Artificial Medicine 1987, Lecture Notes, vol. 33, 1987, pp. 15-24.
29. Weizel J.R., Kerschberg L. A system development methodology for knowledge-based systems. IEEE Trans. Syst., Man Cybern. 19(3):1989;598-605.
30. A. Rossi-Mori, D.M. Pisanelli, F.L. Ricci, The Role of Knowledge Based Systems in Medicine, Epistemological and Evaluation Issues, in: J. Talmon, J. Fox (Eds.), Knowledge-based Systems in Medicine: Methods, Applications and Evaluation, Proceedings of the Workshop 'System Engineering in Medicine', Maastricht, 1989, Lecture Notes in Medical Informatics, vol. 47, Springer, Berlin, 1991, pp. 291-303.
31. R. O'Moore, K. Clarke, J. Brender, P. McNair, P. Nykänen, R. Smeets, J. Talmon, J. Grimson, B. Barber, Methodology for Evaluation of Knowledge-Based Systems, Technical Report of the KAVAS (A1021) AIM Project, vol. EM-1.2, 1990 (public).
32. A. Rector, R. Engelbrecht, T. Diedrich, M. Fieschi, M. Fitter, W. Moser, J. Schneider, Report on Validation and Verification Requirements and Operational Criteria, Technical Report of the EURODIABETA (A1019) AIM Project, vol. Deliverable 12, 1990.
33. Preece A. Towards a methodology for evaluating expert systems. Expert Syst. 7(4):1990;215-223.
34. Wyatt J., Spiegelhalter D. Evaluating medical expert systems: what to test and how? Med. Inform. 15(3):1990;205-217.
35. J. Wyatt, D. Spiegelhalter, Field Trials of Medical Decision Aids: Potential Problems and Solutions, in: P. Clayton (Eds.), Proceeding of the Fifteenth Annual Symposium Computer Applied Medical Care, 1991, pp. 3-7.
36. J. Wyatt, D. Spiegelhalter, Evaluating Medical Expert Systems: What to Test and How?, in: J. Talmon, J. Fox (Eds.), Knowledge-based Systems in Medicine: Methods, Applications and Evaluation, Proceedings of the Workshop 'System Engineering in Medicine', Maastricht, 1989, Lecture Notes in Medical Informatics, vol. 47, Springer, Berlin, 1991, pp. 274-290.
37. Guida G., Mauri G. Evaluating performance and quality of knowledge-based systems: foundation and methodology. IEEE Trans. Knowl. Data Eng. 5(2):1993;204-234.
38. R.T. Plant, The Validation and Verification of Complex Knowledge-based Systems, in: J.A. Wise, V.D. Hopkin, P. Stager (Eds.), Verification and Validation of Complex Systems: Human Factors Issues, Springer, Berlin, 1993, pp. 193-202.
39. Lee S., O'Keefe R.M. Developing a strategy for expert system verification and validation. IEEE Trans. Syst. Man Cybern. 24(4):1994;643-655.
40. Clarke K., O'Moore R., Smeets R., Talmon J., Brender J., McNair P., Nykänen P., Grimson J, Barber B. A methodology for evaluation of knowledge-based systems in medicine. AIM. 6(2):1994;107-121.
41. C.P. Friedman, J.C. Wyatt, Evaluation Methods in Medical Informatics, Springer, Berlin, 1996.
42. S. Hunt, A. Schur, The User Oriented Evaluation Process: A Process for Preserving User Needs during Iterative System Test and Evaluation, Proceedings of the Human Factors Society 35th Annual Meeting 1991, vol. 2, Human Factors Society, Santa Monica, 1991, pp. 1336-1340.
43. Crepea A.T. A systems engineering approach to technology assessment. J. Clin. Eng. 20(4):1995;297-303.
44. C. Ohmann, O. Boy, Q. Yang, A Systematic Approach to the Assessment of User Satisfaction with Health Care Systems: Constructs, Models and Instruments. Medical Informatics Europe '97, Studies in Health Technology and Informatics, in: C. Pappas, N. Maglaveras, J.-R. Scherrer (Eds.), vol. 43, IOS Press, Amsterdam, 1997, pp. 781-785.
45. A. Terazzi, A. Giordano, G. Minuco, An Example of Usability Measurement in Clinical Procedures, Medical Informatics Europe '97, Studies in Health Technology and Informatics, in: C. Pappas, N. Maglaveras, J.-R. Scherrer (Eds.), vol. 43, IOS Press, Amsterdam, 1997, pp. 736-740.
46. M. Demeester, R. Reuscart, The Future is no Longer what it Used to be - Managing Cultural Diversity and Health Telematics, in: J. Brender, J.P. Christensen, J.-R. Scherrer, P. McNair (Eds.), Medical Informatics Europe '96, Studies in Health Technology and Informatics, vol. 34, IOS Press, Amsterdam, 1996, pp. 935-941.
47. J. Brender, Assessment of IT-Based Solutions in Healthcare: Trends in Today's Practice and Justification of Tomorrow's Approach, Medical Informatics Europe '97, Studies in Health Technology and Informatics, in: C. Pappas, N. Maglaveras, J.-R. Scherrer (Eds.), vol. 43, IOS Press, Amsterdam, 1997, pp. 730-735.
48. U. Altmann, V. Haeberlin, A. Tafazzoli, J. Dudeck, Application of a Standard Methodology for the Development of Messages and Aspects of Realization in the Area of Tumor Documentation, in: C. Pappas, N. Maglaveras, J.-R. Scherrer (Eds.), Medical Informatics Europe '97, Studies in Health Technology and Informatics, vol. 43, IOS Press, Amsterdam, 1997, pp. 776-780.
49. G. Braccini, F. Fabbrini, M. Fusani, Software Quality Assessment for Health Care Systems, in: C. Pappas, N. Maglaveras, J.-R. Scherrer (Eds.), Medical Informatics Europe '97, Studies in Health Technology and Informatics, vol. 43, IOS Press, Amsterdam, 1997, pp. 746-750.
50. J.W. van der Hofstede, A.B.W.M. Quak, A.M. van Ginneken, P.W. Macfarlane, J. Watson, P.R.H. Hendriks, C. Zeelenberg, Evaluation Plan for a Cardiological Multi-Media Workstation (I4C Project), in: C. Pappas, N. Maglaveras, J.-R. Scherrer (Eds.), Medical Informatics Europe '97; Studies in Health Technology and Informatics, vol. 43, IOS Press, Amsterdam, 1997, pp. 751-755.
51. Z. Kolitsi, D. Koutsouris, S. Pavlopoulos, N. Pallikarakis, Quality Assurance and Quality Assessment of Health Care Telematic Technologies, in: C. Pappas, N. Maglaveras, J.-R. Scherrer (Eds.), Medical Informatics Europe '97, Studies in Health Technology and Informatics, vol. 43, IOS Press, Amsterdam, 1997, pp. 766-770.
52. G. Lakner, L. Balkányi, G. Surján, J. Kovács, Quality of Healthcare Related Software Applications - Setting up an Accreditation System in Hungary, in: C. Pappas, N. Maglaveras, J.-R. Scherrer (Eds.), Medical Informatics Europe '97, Studies in Health Technology and Informatics, vol. 43, IOS Press, Amsterdam, 1997, pp. 791-795.
53. P. Lauritsen, K. Kaasgaard, Computer Programs and Clinical Contexts, Two Computer Based Exercises for Cognitive rehabilitation, in: C. Pappas, N. Maglaveras, J.-R. Scherrer (Eds.), Medical Informatics Europe '97, Studies in Health Technology and Informatics, vol. 43, IOS Press, Amsterdam, 1997, pp. 756-760.
54. G. Trouessin, B. Barber, The Help of Formal Models for Healthcare Security Policies, in: C. Pappas, N. Maglaveras, J.-R. Scherrer (Eds.), Medical Informatics Europe '97, Studies in Health Technology and Informatics, vol. 43, IOS Press, Amsterdam, 1997, pp. 786-790.
55. M.C. Vissers, J.L. Talmon, Evaluating the MACRO Solution, in: C. Pappas, N. Maglaveras, J.-R. Scherrer (Eds.), Medical Informatics Europe '97, Studies in Health Technology and Informatics, vol. 43, IOS Press, Amsterdam, 1997, pp. 741-745.
56. F. Wiesman, User Evaluation of a Graphical User Interface for Biomedical Literature Search, in: C. Pappas, N. Maglaveras, J.-R. Scherrer (Eds.), Medical Informatics Europe '97, Studies in Health Technology and Informatics, vol. 43, IOS Press, Amsterdam, 1997, pp. 796-800.
57. F.A. Allaert, L. Dusserre, Tele-Expertise: Users' and Suppliers' Liabilities, in: J. Brender, J.P. Christensen, J.-R. Scherrer, P. McNair, Medical Informatics Europe '96, Studies in Health Technology and Informatics, vol. 34, IOS Press, Amsterdam, 1996, pp. 957-961.
58. Walsham G. Management science and organisational change: a framework for analysis. OMEGA Int. J. Manag. Sci. 20(1):1992;1-9.
59. NHS Centre for Coding and Classification, Integrated Clinical Workstation, Specification of User Requirements, Third Draft, NHS CCC, Leicestershire, 1992.
60. P. Docherty, K. Fuchs-Kittowski, P. Kolm, L. Mathiassen (Eds.), System Design for Human Development and Productivity: Participation and Beyond, Proceedings of the IFIP TC 9/WG 9.1 Working Conference 1986, Elsevier, Amsterdam, 1987.
61. T.K. Bikson, J.D. Eveland, Technology Transfer as a Framework for Understanding Social Impacts of Computerization, in: M.J. Smith, G. Salvendy (Eds.), Work With Computers: Organizational, Management, Stress and Health Aspects, Proceedings of the Third Conference on Human-Computer Interaction, vol. 1, Elsevier, Amsterdam, 1989, pp. 28-37.
62. R.I. Benjamin, E. Levison, A Framework for Managing IT - Enabled Change, Sloan Management Review, Summer 1993, pp. 23-33.
63. M. Demeester, Adaptation of KAVAS to the Cultural Components of a Medical Domain, Technical Report of the KAVAS-2 (A2019) AIM Project, vol. TELE-2.1, 1995.
64. J.P. Bansler, E. Havn, The Nature of Software Work: Systems Development as a Labor Process, in: P. van den Besselaar, A. Clement, P. Järvinen (Eds.), Information System, Work and Organization Design, Proceedings of the IFIP TC9/WG9.1 Working Conference on Information System, Work and Organization Design, Elsevier, Amsterdam, 1991, pp. 145-153.
65. J. Stage, The Use of Descriptions in Analysis and Design of Information Systems, in: R.K. Stamper, P. Kerola, R. Lee, K. Lyytinen (Eds.), Proceedings of the IFIP TC8 Working Conference on Collaborative Work, Social Communications and Information Systems, COSCIS '91, Elsevier, Amsterdam, 1991, pp. 237-260.
66. W. Beuschel, What will Impact Assessment tell us about Expert Systems Design?, in: P. van den Besselaar, A. Clement, P. Järvinen (Eds.), Information System, Work and Organization Design; Proceedings of the IFIP TC9/WG9.1 Working Conference on Information System, Work and Organization Design, Elsevier, Amsterdam, 1991, pp. 63-72.
67. M. Ruohonen, Contradictions of Managerial Learning System for Strategic Information Systems Planning, in: R.K. Stamper, P. Kerola, R. Lee, K. Lyytinen (Eds.), Proceedings of the IFIP TC8 Working Conference on Collaborative Work, Social Communications and Information Systems, COSCIS '91, Elsevier, Amsterdam, 1991, pp. 181-211.
68. S. Friis, User Involvement in Requirements Analysis and Specifications, in Theory and Practice, in: P. van den Besselaar, A. Clement, P. Järvinen (Eds.), Information System, Work and Organization Design, Proceedings of the IFIP TC9/WG9.1 Working Conference on Information System, Work and Organization Design, Elsevier, Amsterdam, 1991, pp. 317-325.
69. ISO/IEC, Information Technology-Software Product Evaluation-Quality Characteristics and Guidelines for Their Use, 1st ed., 1991-12-15, vol. ISO 9126-1991 (E), ISO, Genève, 1991.
70. IEEE, IEEE Standard for Software Verification and Validation Plans, vol. 1012-1986, IEEE, New York, 1986.
71. ISO, Quality Management and Quality Assurance Standards - Part 3: Guidelines for the Application of ISO 9001 to the Development, Supply and Maintenance of Software, 1st ed., corrected and reprinted 1993-05-01, vol. ISO 9000-3-1991 (E), ISO, Genève, 1991.
72. IEEE, IEEE Standard for a Software Quality Metrics Methodology, vol. 1061-1992, New York, 1992.
73. IEEE, IEEE Guide for Software Verification and Validation Plans, Vol. 1059-1993, New York, 1994.
74. IEC/TC56/WG 11, Draft IEC 300-3-8 Dependability Management - Part 3: Application Guide, Section 8: Human Reliability, vol. 56/455/CD, IEC, 1995.