Review of health information technology usability study methodologies

Journal of the American Medical Informatics Association

Volumn 19, Issue 3, 2012, Pages 413-422

  Yen, Po Yin ^a   Bakken, Suzanne ^b  

^a OHIO STATE UNIVERSITY   (United States)

^b Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANALYTICAL PARAMETERS; ENVIRONMENTAL IMPACT; HEALTH INFORMATION TECHNOLOGY; HUMAN; INFORMATION TECHNOLOGY; LIFE CYCLE; MEDLINE; QUALITATIVE ANALYSIS; QUANTITATIVE ANALYSIS; REVIEW; SYSTEMATIC REVIEW; TASK PERFORMANCE; THEORY; USABILITY FACTOR;

HUMANS; MEDICAL INFORMATICS; MODELS, THEORETICAL; RESEARCH DESIGN; TECHNOLOGY ASSESSMENT, BIOMEDICAL; USER-COMPUTER INTERFACE;

EID: 84862601178     PISSN: 10675027     EISSN: 1527974X     Source Type: Journal    
DOI: 10.1136/amiajnl-2010-000020     Document Type: Review

References (79)

1. Patterson ES, Cook RI, Render ML. Improving patient safety by identifying side effects from introducing bar coding in medication administration. J Am Med Inform Assoc 2002;9:540-53.
2. Kossman SP. Perceptions of impact of electronic health records on nurses' work. Stud Health Technol Inform 2006;122:337-41.
3. Kushniruk A, Triola M, Stein B, et al. The relationship of usability to medical error: an evaluation of errors associated with usability problems in the use of a handheld application for prescribing medications. Stud Health Technol Inform 2004;107:1073-6.
4. Wyatt JC, Wyatt SM. When and how to evaluate health information systems? Int J Med Inform 2003;69:251-9.
5. ISO 9241-11. Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: guidance on usability. International Organization for Standardization, 1998.
6. Kaufman D, Roberts WD, Merrill J, et al. Applying an evaluation framework for health information system design, development, and implementation. Nurs Res 2006;55(2 Suppl):S37-42.
7. Ammenwerth E, Graber S, Herrmann G, et al. Evaluation of health information systems-problems and challenges. Int J Med Inform 2003;71:125-35.
8. Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004;37:56-76.
9. Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care trends in evaluation research 1982-2002. Methods Inform Med 2005;44:44-56. [erratum appears in Methods Inf Med. 2005;44(3):III].
10. Rahimi B, Vimarlund V. Methods to evaluate health information systems in healthcare settings: a literature review. J Med Syst 2007;31:397-432.
11. Friedman CP, Wyatt J. Evaluation Methods in Biomedical Informatics. 2nd edn. New York, NY: Springer, 2006.
12. Bennett J. Visual Display Terminals: Usability Issues and Health Concerns. Englewood Cliffs New Jersey: Prentice-Hall, 1984.
13. Shackel B. UsabilitydContext, Framework, Definition, Design and Evaluation. In: Shackel B, Richardson SJ, eds. Human Factors for Informatics Usability. New York, NY: Cambridge University Press, 1991:21-37.
14. ISO/IEC 9126. Software engineerting - product quality. International Organization for Standardization, 2001.
15. Abran A, Khelifi A, Suryn W, et al. Usability meanings and interpretations in ISO standards. Software Qual J 2003;11:325-38.
16. Ash JS, Sittig DF, Dykstra RH, et al. Categorizing the unintended sociotechnical consequences of computerized provider order entry. Int J Med Inform 2007;76(Suppl 1):S21-7.
17. Reddy M, Pratt W, Dourish P, et al. Sociotechnical requirements analysis for clinical systems. Meth Inform Med 2003;42:437-44.
18. Shaw R, Coia JE, Michie J. Use of bar code readers and programmable keypads to improve the speed and accuracy of manual data entry in the clinical microbiology laboratory: experience of two laboratories. J Clin Pathol 1999;52:54-60.
19. Henricks WH, Roumina K, Skilton BE, et al. The utility and cost effectiveness of voice recognition technology in surgical pathology. Mod Pathol 2002;15:565-71.
20. Spyglass Consulting Group. Study reveals more than 90% of nurses are reluctant to use tablet PC for bedside. http://www.spyglass-consulting.com/press_releases/SpyglassPR_Mobile_Computing_Nursing.v1.2.pdf 2004.
21. Lanway C, Graham P. Mobile documentation. Wireless PDAs boost job satisfaction for utilization review nurses. Healthc Inform 2003;20:80.
22. Dumas JS, Redish JC. A Practical Guide to Usability Testing. Revised edn. Portland: Intellect Ltd, 1999.
23. Rubin J. Handbook Of Usability Testing. New York: John Wiley & Sons, 1993.
24. Stead WW, Haynes RB, Fuller S, et al. Designing medical informatics research and library-resource projects to increase what is learned. J Am Med Inform Assoc 1994;1:28-33.
25. Gremy F, Degoulet P. Assessment of health information technology: which questions for which systems? Proposal for a taxonomy. Med Inform 1993;18:185-93.
26. Musen MA, Shahar Y, Shortliffe EH. Clinical Decision Support Systems. In: Shortliffe EH, Perreault LE, Wiederhold G, et al, eds. Medical Informatics: Computer Applications in Health Care and Biomedicine. 2nd edn. New York, NY: Springer-Verlag, 2001:573-609.
27. Muller-Staub M, Lavin MA, Needham I, et al. Meeting the criteria of a nursing diagnosis classification: evaluation of ICNP, ICF, NANDA and ZEFP. Int J Nurs Stud 2007;44:702-13.
28. Cho I, Park HA. Development and evaluation of a terminology-based electronic nursing record system. J Biomed Inform 2003;36:304-12.
29. Mihailidis A, Krones L, Boger J. Assistive computing devices: a pilot study to explore nurses' preferences and needs. Comput Inform Nurs 2006;24:328-36.
30. Doran DM, Mylopoulos J, Kushniruk A, et al. Evidence in the palm of your hand: development of an outcomes-focused knowledge translation intervention. Worldviews Evid Based Nurs 2007;4:69-77.
31. Johnson KB, Fitzhenry F. Case report: activity diagrams for integrating electronic prescribing tools into clinical workflow. J Am Med Inform Assoc 2006;13:391-5.
32. Workflow Management Coalition. WfMC: interface 1: process definition interchange - process model. 1998. http://www.wfmc.org/standards/docs/ if19807r3.pdf (accessed 2008).
33. Mueller ML, Ganslandt T, Frankewitsch T, et al. Workflow analysis and evidencebased medicine: towards integration of knowledge-based functions in hospital information systems. Proc AMIA Symp 1999:330-4.
34. Horsky J, Gutnik L, Patel VL. Technology for emergency care: cognitive and workflow considerations. AMIA Annu Symp Proc 2006:344-8.
35. Hyun S, Johnson SB, Bakken S. User-centered approaches for eliciting functional requirements and designing user interfaces: a nursing documentation case study. J Biomed Inform 2008; In press.
36. Effken J, Loeb R, Johnson K, et al. Using cognitive work analysis to design clinical displays. Stud Health Technol Inform 2001;84:127-31.
37. Saleem JJ, Patterson ES, Militello L, et al. Using human factors methods to design a new interface for an electronic medical record. AMIA Annu Symp Proc 2007:640-4.
38. Street AF, Swift K, Annells M, et al. Developing a web-based information resource for palliative care: an action-research inspired approach. BMC Med Inform Decis Mak 2007;7:26.
39. Jalloh OB, Waitman LR. Improving Computerized Provider Order Entry (CPOE) usability by data mining users' queries from access logs. AMIA Annu Symp Proc 2006:379-83.
40. Neve K, Kragh Iversen R, Andersen CK. Is it possible for nurses and doctors to form a useful clinical overview of an EHR? Stud Health Technol Inform 2006;122:314-19.
41. Nielsen J. Usability Engineering. Cambridge: Academic Press, 1993.
42. Thyvalikakath TP, Schleyer TKL, Monaco V. Heuristic evaluation of clinical functions in four practice management systems: a pilot study. J Am Dent Assoc 2007;138:209-10.
43. Cohen T, Kaufman D, White T, et al. Cognitive evaluation of an innovative psychiatric clinical knowledge enhancement system. Stud Health Technol Inform 2004;107:1295-9.
44. Peute LWP, Jaspers MWM. The significance of a usability evaluation of an emerging laboratory order entry system. Int J Med Inform 2007;76:157-68.
45. Elhadad N, McKeown K, Kaufman D, et al. Facilitating physicians' access to information via tailored text summarization. AMIA Annu Symp Proc 2005:226-30.
46. Yu H, Lee M, Kaufman D, et al. Development, implementation, and a cognitive evaluation of a definitional question answering system for physicians. J Biomed Inform 2007;40:236-51.
47. Horsky J, Kaufman DR, Patel VL. The cognitive complexity of a provider order entry interface. AMIA Annu Symp Proc 2003:294-8.
48. Wharton C, Rieman J, Lewis C, et al. The Cognitive Walkthrough Method: A Practitioner's Guide. In: Nielsen J, Mack RL, eds. Usability Inspection Methods. New York: John Wiley & Sons, 1994.
49. Lewis C. Using the "Think Aloud" Method in Cognitive Interface Design. New York: IBM, 1982.
50. Chin JP, Diehl VA, Norman KL. Development of an instrument measuring user satisfaction of the human-computer interface. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 1988:213-18.
51. Norman K, Shneiderman B. Questionnaire for user interaction satisfaction (QUIS). 1997. http://lap.umd.edu/quis/ (accessed 2008).
52. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 1989;13:318-40.
53. Lewis JR. IBM computer usability satisfaction questionnaires - psychometric evaluation and instructions for use. Int J Hum Comput Interact 1995;7:57-78.
54. Yen P, Gorman P, O'Meara K, et al. Usability Testing Of Digital Pen And Paper System In Nursing Documentation. Portland: Department of Biomedical Informatics, Oregon Health & Science University, 2004.
55. Bradshaw KE, Sittig DF, Gardner RM, et al. Latter Day Saints Hospital. Computerbased data entry for nurses in the ICU. MD Comput 1989;6:274-80.
56. Poissant L, Pereira J, Tamblyn R, et al. The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. J Am Med Inform Assoc 2005;12:505-16.
57. Golob JF Jr, Fadlalla AMA, Kan JA, et al. Validation of surgical intensive careinfection registry: a medical informatics system for intensive care unit research, quality of care improvement, and daily patient care. J Am Coll Surg 2008;207:164-73.
58. Rudkin SE, Langdorf MI, Macias D, et al. Personal digital assistants change management more often than paper texts and foster patient confidence. Eur J Emerg Med 2006;13:92-6.
59. Zheng K, Haftel HM, Hirschl RB, et al. Quantifying the impact of health IT implementations on clinical workflow: a new methodological perspective. J Am Med Inform Assoc 2010;17:454-61.
60. Alsos OA. Interaction techniques for using handhelds and PCs together in a clinical setting. NordiCHI, 2006:125-34.
61. Linder JA, Schnipper JL, Tsurikova R, et al. Barriers to electronic health record use during patient visits. AMIA Annu Symp Proc 2006:499-503.
62. Rousseau N, McColl E, Newton J, et al. Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care. BMJ 2003;326:314. [see Comment].
63. Lee TT. Nurses' experiences using a nursing information system: early stage of technology implementation. Comput Inform Nurs 2007;25:294-300.
64. Saleem JJ, Patterson ES, Militello L, et al. Impact of clinical reminder redesign on learnability, efficiency, usability, and workload for ambulatory clinic nurses. J Am Med Inform Assoc 2007;14:632-40. [see Comment].
65. Saleem JJ, Patterson ES, Militello L, et al. Exploring barriers and facilitators to the use of computerized clinical reminders. J Am Med Inform Assoc 2005;12:438-47.
66. Patterson ES, Nguyen AD, Halloran JP, et al. Human factors barriers to the effective use of ten HIV clinical reminders. J Am Med Inform Assoc 2004;11:50-9.
67. Ruland CM, Starren J, Vatne TM. Participatory design with children in the development of a support system for patient-centered care in pediatric oncology. J Biomed Inform 2008;41:624-35.
68. Waller A, Franklin V, Pagliari C, et al. Participatory design of a text message scheduling system to support young people with diabetes. Health Informatics J 2006;12:304-18.
69. Chan W. Increasing the success of physician order entry through human factors engineering. J Healthc Inf Manag 2002;16:71-9.
70. Gennari JH, Reddy M. Participatory design and an eligibility screening tool. Proc AMIA Symp 2000:290-4.
71. Clemensen J, Larsen SB, Kyng M, et al. Participatory design in health sciences: using cooperative experimental methods in developing health services and computer technology. Qual Health Res 2007;17:122-30.
72. Koppel R, Wetterneck T, Telles JL, et al. Workarounds to barcode medication administration systems: their occurrences, causes, and threats to patient safety. J Am Med Inform Assoc 2008;15:408-23.
73. Vogelsmeier AA, Halbesleben JRB, Scott-Cawiezell JR. Technology implementation and workarounds in the nursing home. J Am Med Inform Assoc 2008;15:114-19.
74. Wu SJ, Lehto M, Yih Y, et al. Relationship of estimated resolution time and computerized clinical reminder adherence. AMIA Annu Symp Proc 2007:334-8.
75. Vashitz G, Meyer J, Gilutz H. General practitioners' adherence with clinical reminders for secondary prevention of dyslipidemia. AMIA Annu Symp Proc 2007:766-70.
76. Taylor LK, Tamblyn R. Reasons for physician non-adherence to electronic drug alerts. Stud Health Technol Informat 2004;107:1101-5.
77. Dahlback N, Jonsson A, Ahrenberg L. Wizard of Oz studies - why and how. Knowl Base Syst 1993;6:258-66.
78. Detmer WM, Shiffman S, Wyatt JC, et al. A continuous-speech interface to a decision support system: II. An evaluation using a Wizard-of-Oz experimental paradigm. J Am Med Inform Assoc 1995;2:46-57.
79. Schuler D, Namioka A. Participatory Design: Principles and Practices. Hillsdale, NJ: CRC/Lawrence Erlbaum Associates, 1993.