The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool

Journal of Biomedical Informatics

Volumn 55, Issue , 2015, Pages 1-10

  Peute, Linda W P ^a   de Keizer, Nicolette F ^a   Jaspers, Monique W M ^a  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

Databases; ICU; Methodologies; Think Aloud; Usability; User computer interface

Indexed keywords

DATABASE SYSTEMS; INTENSIVE CARE UNITS; QUALITY CONTROL; SILICON COMPOUNDS; USER EXPERIENCE; USER INTERFACES;

FORMATIVE USABILITY TESTING; HEALTH INFORMATION TECHNOLOGIES; METHODOLOGIES; ORGANIZATION OF INFORMATIONS; THINK ALOUD; USABILITY; USER-COMPUTER INTERFACES; VERBAL PROTOCOL ANALYSIS;

USABILITY ENGINEERING;

ADULT; ARTICLE; BIOINFORMATICS; CLINICAL EFFECTIVENESS; CLINICAL PROTOCOL; COMPARATIVE STUDY; CONCURRENT THINK ALOUD METHOD; FEMALE; HEALTH CARE QUALITY; HUMAN; HUMAN COMPUTER INTERACTION; INTENSIVE CARE UNIT; INTENSIVIST; MALE; PATIENT CARE; PRIORITY JOURNAL; PROGRAM EVALUATION; RETROSPECTIVE THINK ALOUD METHOD; STUDY DESIGN; CLASSIFICATION; CLINICAL PRACTICE; CONSUMER ATTITUDE; DATA MINING; ELECTRONIC HEALTH RECORD; HEALTH PERSONNEL ATTITUDE; MEANINGFUL USE CRITERIA; PHYSICIAN; PROCEDURES; RETROSPECTIVE STUDY; SOFTWARE; SOFTWARE VALIDATION; STATISTICS AND NUMERICAL DATA; UTILIZATION REVIEW;

ATTITUDE OF HEALTH PERSONNEL; CONSUMER BEHAVIOR; DATA MINING; ELECTRONIC HEALTH RECORDS; MEANINGFUL USE; PHYSICIANS; PRACTICE PATTERNS, PHYSICIANS'; RETROSPECTIVE STUDIES; SOFTWARE; SOFTWARE VALIDATION; UTILIZATION REVIEW;

EID: 84930694827     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2015.02.006     Document Type: Article

References (40)

1. Kohn LT et al. editors. To Err is human: building a safer health system. Institute of Medicine. <>. http://newton.nap.edu/catalog/9728.html.
2. Campbell EM, Sittig DF. Guappone KP, Dykstra RH, Ash JS. Overdependence on technology: an unintended adverse consequence of computerized provider order entry. In: AMIA annu symp proc. 2007. p. 94-8.
3. National questionnaire study on clinical ICT systems proofs: Physicians suffer from poor usability. Johanna Viitanen, Hannele Hyppönen, Tinja Lääveri, Jukka Vänskä, Jarmo Reponen, Ilkka Winblad. Int J Med Inf 2011; 80(10):708-25.
4. Horsky J., Zhang J., Patel V.L. To err is not entirely human: complex technology and user cognition. J Biomed Inform 2005, 38(4):264-266.
5. ISO 9241-11. International standard: ISO 9241 ergonomic requirements for office work with visual display terminals, Part 11: Guidance on usability. Geneve: International Organization for Standardization; 1996.
6. Dix A., Finlay J., Abowd G., Beale R. Human-computer interaction 2003, Prentice Hall, Berlin, [date last visited: April 8th, 2011)]. 3rd ed.
7. Svanæs D., Alsos O.A., Dahl Y. Usability testing of mobile ICT for clinical settings: methodological and practical challenges. Int J Med Inform 2010, 79:e24-e34.
8. Lai T.-Y. Iterative refinement of a tailored system for self-care management of depressive symptoms in people living with HIV/AIDS through heuristic evaluation and end user testing. Int J Med Inform 2007, 76(SUPPL):2.
9. Khajouei R., Hasman A., Jaspers M.W.M. Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system. Int J Med Inform 2011, 80(5):341-350.
10. Norio Ishii, Kazuhisa Miwa. Interactive processes between mental and external operations in creative activity: a comparison of experts' and novices' performance. In: Proceedings of the 4th conference on creativity & cognition. 2002 Oct. 13-16. p. 178-85.
11. Nielsen J. Novice vs. expert users. Alertbox, February 6, 2000. <. http://www.useit.com/alertbox/20000206.html.
12. Kjeldskov J., Skov M.B., Stage J. A longitudinal study of usability in health care: does time heal?. Int J Med Inform 2010, 79(6):e135-e143.
13. Ericsson K.A., Simon H.A. Protocol analysis: verbal protocols as data 1993, MIT Press, Cambridge.
14. Boren M.T., Ramey J. Thinking aloud: reconciling theory and practice. IEEE Trans Prof Comm 2000, 43:261-278.
15. Clayton LH. Using the think aloud method in cognitive interface design. IBM RC-9265.1982. Yorktown heights. NY: IBM Thomas J Watson Research Center.
16. Bowers VA, Snyder HL. Concurrent versus retrospective verbal protocols for comparing window usability. Human factors society 34th meeting. 1990 Oct. 8-12 (Santa Monica: HFES). p. 1270-1274.
17. De Jong M., Schellens P.J. Towards a document evaluation methodology: what does research tell us about the validity and reliability of methods?. IEEE Trans Prof Commun 2000, 43:242-260.
18. Russo J.E., Johnson E.J., Stephens D.L. The validity of verbal protocols. Mem Cognition 1989, 17:759-769.
19. Teague R, De Jesus K, Nunes-Ueno M. Concurrent vs. post-task usability test ratings. In: Conference on human factors and computing systems, 2001 31 March-5 April (Seattle, WA: ACM SIGCHI):289-290.
20. Hartson H.R., Andre T.S., Williges R.C. Criteria for evaluating usability evaluation methods. Int J Hum Compos Interact 2001, 13(4):241-444.
21. Arts D., de Keizer N., Scheffer G.J., de Jonge E. Quality of data collected for severity of illness scores in the Dutch National Intensive Care Evaluation (NICE) registry. Intens Care Med 2002, 28(5):656-659.
22. De Keizer N.F., Peute L.W., Van der Zwan E., Jaspers M.W., De Jonge E. NICE online; a web-based tool for monitoring performance measures in intensive care. Neth J Crit Care 2011 June, 15(3):131-136.
23. Kushniruk A.W., Patel V.L. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004, 37:56-76.
24. Nielsen J. Severity ratings for usability problems. ; 2010 <> [cited 10 A.D. Oct 24 2011]. http://useit.com/papers/heuristic/severityrating.html.
25. Gwet KL. Handbook of inter-rater reliability, 2nd ed.; 2010. (Gaithersburg: Advanced Analytics, LLC) ISBN 978-0970806222.
26. Sears A. Heuristic walkthroughs: finding the problems without the noise. IJHCI 1997, 9:213-234.
27. Turner C.W., Lewis J.R., Nielsen J. Determining usability test sample size. International encyclopedia of ergonomics and human factors 2006, vol. 3, 2nd ed.. CRC Press, Boca Raton (FL).
28. Payne S.J. Users' mental models: the very ideas. HCI models, theories, and frameworks 2003, 135-154. Morgan Kaufman Publisherspp, San Francisco. M. John (Ed.).
29. Norman D.A. The design of everyday things 1988, Doubleday, New York.
30. Nielsen J. Getting access to what goes on in people's heads?: reflections on the think-aloud technique. NordiCHI '02 Proceedings of the second Nordic conference on Human-computer interaction, p: 101-110.
31. Kuusela H., Paul P. A comparison of concurrent and retrospective verbal protocol analysis. Am J Psychol 2000, 113(3):387-404. [Fall].
32. Johnson-Laird P.N. Mental models: towards a cognitive science of language, inference, and consciousness 1983, Cambridge University Press, Cambridge.
33. Haak, van den M.J., Jong de M.D.T., Schellens P.J. Retrospective versus concurrent think-aloud protocols: testing the usability of an online library catalogue. Behav Inform Technol 2003, 22(5):339-351.
34. McDonald S., Zhao T., Edwards H.M. Dual verbal elicitation: the complementary use of concurrent and retrospective reporting within a usability test. Int J Human Comput Interact 2013, 29(10).
35. Spires H.A., Donley J. Prior knowledge activation: inducing engagement with informational texts. J Educ Psychol 1998, 90(2):249-260.
36. Shertz J., Weiser M. A study of programming problem representation in novice and expert programmers. Proceedings of the eighteenth annual computer personnel research conference (SIGCPR '81) 1981, 302-322. ACM, New York (NY, USA). M. Awad Elias (Ed.).
37. Muin M., Fontelo P., Liu F., Ackerman M. SLIM: an alternative Web interface for MEDLINE/PubMed searches - a preliminary study. BMC Med Inform Decis Making 2005 Dec, 1(5):37.
38. Horvath M.M., Winfield S., Evans S., Slopek S., Shang H., Ferranti J. The DEDUCE Guided Query tool: Providing simplified access to clinical data for research and quality improvement. J Biomed Inform 2011, 44(2):266-276.
39. Nielsen J. Usability engineering at a discount. Designing and using human-computer interfaces and knowledge based systems 1989, 394-401. Elsevier Science Publishers, Amsterdam. G. Salvendy, M.J. Smith (Eds.).
40. Guan Z., Lee S., Cuddihy E., Ramey J. The validity of the stimulated retrospective think-aloud method as measured by eye tracking. Proceedings of the SIGCHI conference on human factors in computing systems (CHI '06) 2006, 1253-1262. ACM, New York (NY, USA). R. Grinter, T. Rodden, P. Aoki, E. Cutrell, R. Jeffries, O. Gary (Eds.).