With how many users should you test a medical infusion pump? Sampling strategies for usability tests on high-risk systems

Journal of Biomedical Informatics

Volumn 46, Issue 4, 2013, Pages 626-641

  Schmettow, Martin ^a   Vos, Wendy ^b   Schraagen, Jan Maarten ^c  

^a UNIVERSITY OF TWENTE   (Netherlands)

^b Arbeidsinspectie Industrie   (Netherlands)

^c TNO   (Netherlands)

Author keywords

Ergonomics; Infusion pump; Patient safety; Sample size; Usability; User testing

Indexed keywords

INFUSION PUMP; PATIENT SAFETY; SAMPLE SIZES; USABILITY; USER TESTING;

BIOMEDICAL EQUIPMENT; ERGONOMICS; PUMPS; SAFETY TESTING; SAMPLING;

USABILITY ENGINEERING;

ADULT; ARTICLE; AWARENESS; CLINICAL ARTICLE; CLINICAL EFFECTIVENESS; COMPUTER SIMULATION; CONTROL STRATEGY; CONTROLLED STUDY; DIAGNOSTIC TEST; FEMALE; FOOD AND DRUG ADMINISTRATION; GEOMETRY; HAZARD RATIO; HEALTH CARE QUALITY; HIGH RISK PATIENT; HUMAN; INFUSION PUMP; INTENSIVE CARE UNIT; MALE; MATHEMATICAL MODEL; MEDICAL DEVICE; MULTICENTER STUDY; PATIENT MONITORING; PATIENT SAFETY; PHYSICIAN; PRACTICE GUIDELINE; PREDICTION; PRIORITY JOURNAL; PROBLEM IDENTIFICATION; QUALITY CONTROL; SAMPLE SIZE; STATISTICAL MODEL; USABILITY TEST;

EID: 84880782123     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2013.04.007     Document Type: Article

References (70)

1. Obradovich J.H., Woods D.D. Users as designers: how people cope with poor HCI design in computer-based medical devices. Hum Factors 1996, 38:574-592.
2. Lin L., Isla R., Doniz K., Harkness H., Vicente K.J., Doyle D.J. Applying human factors to the design of medical equipment: patient-controlled analgesia. J Clin Monit Comput 1998, 14:253-263.
3. Schmettow M. Sample size in usability studies. Commun ACM 2012, 55:64.
4. Schmettow M. Controlling the usability evaluation process under varying defect visibility. In: BCS-HCI '09: Proceedings of the 23rd British HCI group annual conference on people and computers: celebrating people and technology. Swinton, UK: British Computer Society; 2009. p. 188-97.
5. Gartner K., Liljegren E., Osvalder A.L., Dahlman S. Arguing for the need of triangulation and iteration when designing medical equipment. Nature of requirements on medical equipment derived from a human factors approach and from usability tests: methodological aspects. J Clin Monit Comput 2002, 17:105-114.
6. Garmer K. Application of usability testing to the development of medical equipment. Usability testing of a frequently used infusion pump and a new user interface for an infusion pump developed with a Human Factors approach. Int J Ind Ergon 2002, 29:145-159.
7. Liljegren E, Osvalder AL, Dahlman S. Setting the requirements for a user-friendly infusion pump. In: Proceedings of the human factors and ergonomics society annual meeting. SAGE Publications; 2000. p. 132-5.
8. Sawyer D, Aziz KJ, Backinger CL, Beers ET, Lowery A, Sykes SM. Do it by design, an introduction to human factors in medical devices. US Department of Health and Human Services, Public Health Service, Food and Drug Administration, Center for Devices and Radiological Health; 1996.
9. Cook R.I., Woods D.D., Howie M.B., Horrow J.C., Gaba D.M. Case 2 - 1992 unintentional delivery of vasoactive drugs with an electromechanical infusion device. J Cardiothorac Vasc Anesth 1992, 6:238-244.
10. Hicks R.W., Sikirica V., Nelson W., Schein J.R., Cousins D.D. Medication errors involving patient-controlled analgesia. Am J Health Syst Pharm 2008, 65:429-440.
11. Cohen M.R. Preventing errors associated with PCA pumps. Nursing 1993, 23.
12. Department of Health. Chief Pharmacist's report. Building a safer NHS for patients: improving medication safety, London, UK; 2004.
13. Leape L.L. Error in medicine. JAMA 1994, 272:1851-1857.
14. Vincent C., Neale G., Woloshynowych M. Adverse events in British hospitals: preliminary retrospective record review. BMJ 2001, 322:517-519.
15. Webb R.K., Currie M., Morgan C.A., Williamson J.A., Mackay P., Russell W.J., et al. The Australian incident monitoring study: an analysis of 2000 incident reports. Anaesth Intensive Care 1993, 21:520-528.
16. Vicente K.J., Kada-Bekhaled K., Hillel G., Cassano A., a Orser B. Programming errors contribute to death from patient-controlled analgesia: case report and estimate of probability. Can J Anaesth 2003, 50:328-332.
17. Martin J.L., Norris B.J., Murphy E., Crowe J.A. Medical device development: the challenge for ergonomics. Appl Ergon 2008, 39:271-283.
18. Association of advancement of medical instrumentation. ANSI/AAMI HE75; 2009 (n.d.).
19. Sarter N.B., Woods D.D. How in the world did we ever get into that mode? Mode error and awareness in supervisory control, human factors. J Hum Factors Ergon Soc 1995, 37:5-19.
20. Liljegren E. Usability in a medical technology context assessment of methods for usability evaluation of medical equipment. Int J Ind Ergon 2006, 36:345-352.
21. Center for Devices and Radiological Health U.S. Food and Drug Administration. Infusion pump improvement initiative (n.d.).
22. Kaye R., Crowley J. Medical device use-safety: incorporating human factors engineering into risk management. Hum Factors 2000.
23. Bastien J.M.C. Usability testing: a review of some methodological and technical aspects of the method. Int J Med Inform 2010, 79:e18-e23.
24. Bevan N., Barnum C., Cockton G., Nielsen J., Spool J., Wixon D. The "magic number 5": is it enough for web testing? 2003, ACM, New York (NY), USA.
25. Nielsen J. Why you only need to test with 5 users; 2000. <>http://www.useit.com/alertbox/20000319.html.
26. Hwang W., Salvendy G. Number of people required for usability evaluation: the 10±2 rule. Commun ACM 2010, 53:130-133.
27. Molich R. How many participants needed to test usability?. Commun ACM 2010, 53:7.
28. European Committee for Electrotechnical Standardization. Medical devices - application of usability engineering to medical devices; 2008. p. 110.
29. Boehm B.W., Basili V.R. Software defect reduction top 10 list. IEEE Comput 2001, 34:135-137.
30. Lewis J.R. Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. Int J Hum-Comput Interact 2001, 13:445-479.
31. Virzi R. Refining the test phase of usability evaluation: how many subjects is enough?. Hum Factors: J Hum Factors Ergon Soc 1992, 34:457-468.
32. Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. in: ACM Press (Ed.), Proceedings of INTERCHI 1993. New York (NY), USA: ACM Press; 1993. p. 206-13.
33. Hwang W., Salvendy G. Integration of usability evaluation studies via a novel meta-analytic approach: what are significant attributes for effective evaluation?. Int J Hum-Comput Interact 2009, 25:282.
34. Schmettow M. Heterogeneity in the usability evaluation process. In: BCS-HCI '08: proceedings of the 22nd British HCI group annual conference on HCI 2008. Swinton, UK: British Computer Society; 2008. p. 89-98.
35. Caulton D.A. Relaxing the homogeneity assumption in usability testing. Behav Inf Technol 2001, 20:1-7.
36. Woolrych A, Cockton G, Hindmarch M. Falsification testing for usability inspection method assessment. In: Proceedings of the {HCI04} conference on people and computers XVIII; 2004.
37. Turner C.W., Lewis J.R., Nielsen J. Determining usability test sample size. International encyclopedia of ergonomics and human factors 2006, 3084-3088. CRC Press, Boca Raton (FL). 2nd ed. W. Karwowski (Ed.).
38. Finstad K. Analogical problem solving in casual and experienced users: when interface consistency leads to inappropriate transfer. Hum-Comput Interact 2008, 23:381-405.
39. Carroll J.M., Rosson M.B. Paradox of the active user. Interfacing thought 1987, 80-111. MIT Press. J.M. Carroll (Ed.).
40. U.S.Department of Health and Human Services Draft guidance for industry and food and drug administration staff applying human factors and usability engineering to optimize medical device design. Hum Factors 2011, 7.
41. Kohn L.T., Corrigan J.M., Donaldson M.S. To err is human: Building a Safer Health System 2000, National Academies Press, Washington, DC.
42. Beydon L., Conreux F., Le Gall R., Safran D., Cazalaa J.B. Analysis of the French health ministry's national register of incidents involving medical devices in anaesthesia and intensive care. Br J Anaesth 2001, 86:382-387.
43. National Patient Safety Agency. National reporting and learning service. Design for patient safety: a guide to the design of electronic infusion devices; 2010.
44. Lin L., Vicente K.J., Doyle D.J.J. Patient safety, potential adverse drug events, and medical device design: a human factors engineering approach. J Biomed Inform 2001, 34:274-284.
45. Gray W.D., Salzman M.C. Damaged merchandise? A review of experiments that compare usability evaluation methods. Hum-Comput Interact 1998, 13:203.
46. Lynch S.M. Introduction to applied bayesian statistics and estimation for social scientists 2007, Springer, New York (NY).
47. Frederic P., Lad F. Two moments of the logitnormal distribution. Commun Stat - Simul Comput 2008, 37:1263-1269.
48. Schmettow M, Vietze W. Introducing item response theory for measuring usability inspection processes. In: Proceeding of the twenty-sixth annual CHI conference on human factors in computing systems - CHI '08. New York, USA: ACM Press; 2008. p. 893.
49. Rubin Jeffrey, Chisnell Dana Handbook of usability testing: how to plan, design, and conduct effective tests 2008, John Wiley & Sons. 2nd ed.
50. Molich R., Dumas J. Comparative usability evaluation (CUE-4). Behav Inf Technol 2008, 27:263-281.
51. Sears A. Heuristic walkthroughs: finding the problems without the noise. Int J Hum-Comput Interact 1997, 9:213-234.
52. R Development Core Team R: A language and environment for statistical computing 2011, R Foundation for Statistical Computing, Vienna, Austria.
53. Kanis H. Estimating the number of usability problems. Appl Ergon 2010, 42:337-347.
54. Schmettow M, Niebuhr S. A pattern-based usability inspection method: first empirical performance measures and future issues. In: Ramduny-Ellis D, Rachovides D, editors. BCS HCI '07: Proceedings of the 2007 British computer society conference on human-computer interaction; 2007. p. 99-102.
55. Schmettow M., Bach C., Scapin D.L. Effizientere Usability Evaluationen mit gemischten Prozessen. Mensch Und Computer 2010: Interaktive Kulturen 2010, 271-280. Oldenbourg Verlag, München. J. Ziegler, A. Schmidt (Eds.).
56. Faulkner L. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods 2003, 35:379-383.
57. Nielsen J. How many test users in a usability study? Jacob Nielsen's Alertbox, Bi-weekly Column; 2012.
58. Sauro J. Why you only need to test with five users (explained). Weblog measuring usability; 2011.
59. Lewis J.R. Sample sizes for usability studies: additional considerations. Hum Factors 1994, 36:368-378.
60. Taleb N.N. The black swan 2007, Random House.
61. John B.E., Marks S.J. Tracking the effectiveness of usability evaluation methods. Behav Inf Technol 1997, 16:188-202.
62. Hertzum M., Jacobsen N.E. The evaluator effect: a chilling fact about usability evaluation methods. Int J Hum-Comput Interact 2001, 13:421-443.
63. Hornbæk K., Frøkjær E. Comparison of techniques for matching of usability problem descriptions. Interact Comput 2008, 20:505-514.
64. Molich R., Ede M.R., Kaasgaard K., Karyukin B. Comparative usability evaluation. Behav Inf Technol 2004, 23:65.
65. Rivest L.-P. Why a time effect often has a limited impact on capture-recapture estimates in closed populations. Can J Stat 2008, 36:75-84.
66. Lindgaard G, Chattratichart J. Usability testing - what have we overlooked? In: Proceedings of the SIGCHI conference on human factors in computing systems - CHI '07. New York: ACM Press; 2007. p. 1415.
67. Hahsler M., Hornik K. arules - A computational environment for mining association rules and frequent item sets. J Od Stat Softw 2005, 14.
68. Cahoy D., Phoha V. Model dependence in sample-size calculation. Commun ACM 2012, 55:7.
69. Winkelmann R. Unobserved heterogeneity. Econometric analysis of count data 2008, 127-142. Springer, Berlin, Heidelberg.
70. Burnham K.P., Anderson D.R. Multimodel inference. Understanding AIC and BIC in model selection. Sociol Methods Res 2004, 33:261-304.