Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR

International Journal of Medical Informatics

Volumn 83, Issue 5, 2014, Pages 361-367

  Walji, Muhammad F ^a   Kalenderian, Elsbeth ^b   Piotrowski, Mark ^a   Tran, Duong ^a   Kookal, Krishna K ^a   Tokede, Oluwabunmi ^b   White, Joel M ^c   Vaderhobli, Ram ^c   Ramoni, Rachel ^d   Stark, Paul C ^e   Kimmes, Nicole S ^f   Lagerweij, Maxim ^g   Patel, Vimla L ^h  

^a UNIVERSITY OF TEXAS   (United States)

^b HARVARD SCHOOL OF DENTAL MEDICINE   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

^d HARVARD MEDICAL SCHOOL   (United States)

^e TUFTS UNIVERSITY   (United States)

^f CREIGHTON UNIVERSITY   (United States)

^g UNIVERSITY OF AMSTERDAM   (Netherlands)

^h NEW YORK ACADEMY OF MEDICINE   (United States)

Author keywords

EHR; Human factors; Methodology; Usability

Indexed keywords

HUMAN ENGINEERING; INFORMATION TECHNOLOGY; TEACHING; TESTING; USABILITY ENGINEERING;

COMPARATIVE ASSESSMENT; COMPREHENSIVE EVALUATION; EHR; HEALTH INFORMATION TECHNOLOGIES; METHODOLOGY; SEMI STRUCTURED INTERVIEWS; USABILITY; USABILITY EVALUATION METHODS;

SURVEYS;

ARTICLE; COMPUTER PROGRAM; DENTAL EDUCATION; DENTAL STUDENT; ELECTRONIC MEDICAL RECORD; EVALUATION STUDY; FEMALE; HEALTH CARE SURVEY; HUMAN; MALE; PRIORITY JOURNAL; RESIDENT; SEMI STRUCTURED INTERVIEW; UNIVERSITY; COMPARATIVE STUDY; COMPUTER INTERFACE; INFORMATION PROCESSING; INTERVIEW; MEDICAL INFORMATICS; ONLINE SYSTEM; STANDARDS; UTILIZATION; UTILIZATION REVIEW;

DATA COLLECTION; ELECTRONIC HEALTH RECORDS; FEMALE; HUMANS; INTERVIEWS AS TOPIC; MALE; MEDICAL INFORMATICS; ONLINE SYSTEMS; SOFTWARE; USER-COMPUTER INTERFACE; UTILIZATION REVIEW;

EID: 84897389158     PISSN: 13865056     EISSN: 18728243     Source Type: Journal    
DOI: 10.1016/j.ijmedinf.2014.01.010     Document Type: Article

References (48)

1. Blumenthal D., Glaser J.P. Information technology comes to medicine. N. Engl. J. Med. 2007, 356(24):2527-2534.
2. Chaudhry B., et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann. Intern. Med. 2006, 144(10):742-752.
3. Hillestad R., et al. Can electronic medical record systems transform health care? Potential health benefits: savings, and costs. Health Aff. (Millwood) 2005, 24(5):1103-1117.
4. Haux R. Health information systems - past, present, future. Int. J. Med. Inf. 2006, 75(3-4):268-281.
5. Medicine, I.o. Health IT and Patient Safety Building Safer Systems for Better Care 2011, Available from: http://www.iom.edu/~/media/Files/Report Files/2011/Health-IT/HealthITandPatientSafetyreportbrieffinal_new.pdf (cited 16.07.12).
6. Patel V.L., et al. Translational cognition for decision support in critical care environments: a review. J. Biomed. Inform. 2008, 41(3):413-431.
7. Zhang J. Human-centered computing in health information systems. Part 1: analysis and design. J. Biomed. Inform. 2005, 38(1):1-3.
8. Zhang J. Human-centered computing in health information systems part 2: evaluation. J. Biomed. Inform. 2005, 38(3):173-175.
9. Zhang J., Walji M.F. TURF. Toward a unified framework of EHR usability. J. Biomed. Inform. 2011.
10. Bates D.W., et al. Reducing the frequency of errors in medicine using information technology. J. Am. Med. Inform. Assoc. 2001, 8(4):299-308.
11. Ash J.S., Berg M., Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J. Am. Med. Inform. Assoc. 2004, 11(2):104-112.
12. Thyvalikakath T.P., et al. A usability evaluation of four commercial dental computer-based patient record systems. J. Am. Dent. Assoc. 2008, 139(12):1632-1642.
13. Peute L.W., Jaspers M.W. The significance of a usability evaluation of an emerging laboratory order entry system. Int. J. Med. Inf. 2007, 76(2-3):157-168.
14. Campbell E.M., et al. Types of unintended consequences related to computerized provider order entry. J. Am. Med. Inform. Assoc. 2006, 13(5):547-556.
15. Kushniruk A.W., et al. Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int. J. Med. Inf. 2005, 74(7-8):519-526.
16. Koppel R., et al. Role of computerized physician order entry systems in facilitating medication errors. J. Am. Med. Assoc. 2005, 293(10):1197-1203.
17. Horsky J., Zhang J., Patel V.L. To err is not entirely human: complex technology and user cognition. J. Biomed. Inform. 2005, 38(4):264-266.
18. Horsky J., Kuperman G.J., Patel V.L. Comprehensive analysis of a medication dosing error related to CPOE. J. Am. Med. Inform. Assoc. 2005, 12(4):377-382.
19. Han Y.Y., et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005, 116(6):1506-1512.
20. Middleton B., et al. Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA. J. Am. Med. Inform. Assoc. 2013.
21. Nykanen P., et al. Guideline for good evaluation practice in health informatics (GEP-HI). Int. J. Med. Inf. 2011, 80(12):815-827.
22. Yusof M.M., et al. Investigating evaluation frameworks for health information systems. Int. J. Med. Inf. 2008, 77(6):377-385.
23. Lowry S.Z., et al. Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Record 2012, 780. National Institute of Standards and Technology, NISTIR.
24. Test Procedure for §170.314(g)(3) Safety-enhanced Design 2013, Available from: http://www.healthit.gov/sites/default/files/170.314g3safetyenhanceddesign_2014_tp_approved_v1.3_0.pdf (cited 12.06.13).
25. Rose A.F., et al. Using qualitative studies to improve the usability of an EMR. J. Biomed. Inform. 2005, 38(1):51-60.
26. Khajouei R., Hasman A., Jaspers M.W. Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system. Int. J. Med. Inf. 2011, 80(5):341-350.
27. Kushniruk A., Patel V. Cognitive and usability engineering methods for the evaluation of clinical information systems. J. Biomed. Inform. 2004, 37(1):56-76.
28. Chan J., et al. Usability evaluation of order sets in a computerised provider order entry system. BMJ Qual. Saf. 2011, 20(11):932-940.
29. Johnson C.M., et al. Can prospective usability evaluation predict data errors?. AMIA Annu. Symp. Proc. 2010, 2010:346-350.
30. Khajouei R., et al. Classification and prioritization of usability problems using an augmented classification scheme. J. Biomed. Inform. 2011, 44(6):948-957.
31. Khajouei R., et al. Classification and prioritization of usability problems using an augmented classification scheme. J. Biomed. Inform. 2011.
32. Saitwal H., et al. Assessing performance of an Electronic Health Record (EHR) using Cognitive Task Analysis. Int. J. Med. Inf. 2010, 79(7):501-506.
33. Khajouei R., et al. Effect of predefined order sets and usability problems on efficiency of computerized medication ordering. Int. J. Med. Inf. 2010, 79(10):690-698.
34. Viitanen J., et al. National questionnaire study on clinical ICT systems proofs: physicians suffer from poor usability. Int. J. Med. Inf. 2011, 80(10):708-725.
35. Jaspers M.W. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int. J. Med. Inf. 2009, 78(5):340-353.
36. Peleg M., et al. Using multi-perspective methodologies to study users' interactions with the prototype front end of a guideline-based decision support system for diabetic foot care. Int. J. Med. Inf. 2009, 78(7):482-493.
37. Horsky J., et al. Complementary methods of system usability evaluation: surveys and observations during software design and development cycles. J. Biomed. Inform. 2010, 43(5):782-790.
38. Walji M.F., et al. Detection and characterization of usability problems in structured data entry interfaces in dentistry. Int. J. Med. Inf. 2012.
39. Kalenderian E., et al. The development of a dental diagnostic terminology. J. Dent. Educ. 2011, 75(1):68-76.
40. Card S.K., Moran T.P., Newell A. The Psychology of Human-Computer Interaction 1983, Lawrence Erbaum Associates, London.
41. John B., Prevas K., Salvucci D., Koedinger K. Predictive Human Performance Modeling Made Easy. Proceedings of CHI 2004.
42. Hill H.K., Stewart D.C., Ash J.S. Health Information Technology Systems profoundly impact users: a case study in a dental school. J. Dent. Educ. 2010, 74(4):434-445.
43. Menachemi N., Langley A., Brooks R.G. The use of information technologies among rural and urban physicians in Florida. J. Med. Syst. 2007, 31(6):483-488.
44. Loomis G.A., et al. If electronic medical records are so great, why aren't family physicians using them?. J. Fam. Pract. 2002, 51(7):636-641.
45. Valdes I., et al. Barriers to proliferation of electronic medical records. Inform. Prim. Care 2004, 12(1):3-9.
46. Kemper A.R., Uren R.L., Clark S.J. Adoption of electronic health records in primary care pediatric practices. Pediatrics 2006, 118(1):e20-e24.
47. Laerum H., Ellingsen G., Faxvaag A. Doctors' use of electronic medical records systems in hospitals: cross sectional survey. Br. Med. J. 2001, 323(7325):1344-1348.
48. Ludwick D.A., Doucette J. Primary Care physicians' experience with electronic medical records: barriers to implementation in a fee-for-service environment. Int. J. Telemed. Appl. 2009, 2009:853524.