Scrutinising usability evaluation: Does thinking aloud affect behaviour and mental workload?

Behaviour and Information Technology

Volumn 28, Issue 2, 2009, Pages 165-181

  Hertzum, Morten ^a   Hansen, Kristin D ^a,b   Andersen, Hans H K ^b,c  

^a ROSKILDE UNIVERSITY   (Denmark)

^b TECHNICAL UNIVERSITY OF DENMARK   (Denmark)

^c NATIONAL RESEARCH CENTRE FOR THE WORKING ENVIRONMENT   (Denmark)

Author keywords

Thinking aloud; Usability evaluation methods; User testing; Validity; Verbalisation

Indexed keywords

THINKING ALOUD; USABILITY EVALUATION METHODS; USER TESTING; VALIDITY; VERBALISATION;

EID: 61349127910     PISSN: 0144929X     EISSN: 13623001     Source Type: Journal    
DOI: 10.1080/01449290701773842     Document Type: Article

References (42)

1. Berry, D.C. and Broadbent, D.E., 1990. The role of instruction and verbalisation in improving performance on complex search tasks. Behaviour & Information Technology, 9 (3), 175-190.
2. Boren, T. and Ramey, J., 2000. Thinking aloud: reconciling theory and practice. IEEE Transactions on Professional Communication, 43 (3), 261-278.
3. Campbell, D.J., 1988. Task complexity: a review and analysis. Academy of Management Review, 13 (1), 40-52.
4. Chi, M.T.H., Bassok, M., Lewis, M., Reimann, P., and Glaser, R., 1989. Self-explanations: how students study and use examples in learning to solve problems. Cognitive Science, 13 (2), 145-182.
5. Chi, M.T.H., De Leeuw, N., Chiu, M.-H., and LaVancher, C, 1994. Eliciting self-explanations improves understanding. Cognitive Science, 18 (3), 439-477.
6. Cockton, G. and Woolrych, A., 2002. Sale must end: should discount methods be cleared off HCI's shelves? ACM Interactions, 9 (5), 13-18.
7. DeShon, R.P., Chan, D., and Weissbein, D.A., 1995. Verbal overshadowing effects on Raven's advanced progressive matrices: evidence for multidimensional performance determinants. Intelligence, 21 (2), 135-155.
8. Dumas, J.S., 2003. User-based evaluations. In: J. Jacko and A. Sears, eds. The human-computer interaction handbook. Mahwah, NJ: Erlbaum, 1093-1117.
9. Dumas, J.S. and Redish, J.C., 1999. A practical guide to usability testing. Revised edition. Exeter, UK: Intellect.
10. Ericsson, K.A. and Simon, H.A., 1980. Verbal reports as data. Psychological Review, 87 (3), 215-251.
11. Ericsson, K.A. and Simon, H.A., 1993. Protocol analysis: verbal reports as data. Revised edition. Cambridge, MA: MIT Press.
12. Frøkjær, E. and Hornbæk, K., 2005. Cooperative usability testing: complementing usability tests with usersupported interpretation sessions. In: Extended abstracts of the CHI 2005 conference on human factors in computing systems, 2-5 April, Portland, Oregon. New York: ACM Press, 1383-1386.
13. Goldberg, J.H. and Kotval, X.P., 1999. Computer interface evaluation using eye movements: methods and constructs. International Journal of Industrial Ergonomics, 24 (6), 631-645.
14. Guan, Z., Lee, S., Cuddihy, E., and Ramey, J., 2006. The validity of the stimulated retrospective think-aloud method as measured by eye tracking. In: Proceedings of the CHI 2006 conference on human factors in computing systems, 22-27 April, Montreal, CA. New York: ACM Press, 1253-1262.
15. Hart, S.G. and Staveland, L.E., 1988. Development of NASA-TLX (task load index): results of empirical and theoretical research. In: P.A. Hancock and N. Meshkati, eds. Human mental workload. Amsterdam: Elsevier, 139-183.
16. Held, J.E. and Biers, D.W., 1992. Software usability testing: do evaluator intervention and task structure make any difference? In: Proceedings of the human factors society 36th annual meeting, 12-16 October, Atlanta, GA. Santa Monica, CA: HFS, 1215-1219.
17. Hendy, K.C., Hamilton, K.M., and Landry, L.N., 1993. Measuring subjective workload: when is one scale better than many? Human Factors, 35 (4), 579-601.
18. James, W., 1890. The principles of psychology. New York: Holt.
19. Just, M.A. and Carpenter, P.A., 1980. A theory of reading: from eye fixations to comprehension. Psychological Review, 87 (4), 329-354.
20. Krahmer, E. and Ummelen, N., 2004. Thinking about thinking aloud: a comparison of two verbal protocols for usability testing. IEEE Transactions on Professional Communication, 47 (2), 105-117.
21. Landis, J.R. and Koch, G.G., 1977. The measurement of observer agreement for categorical data. Biometrics, 33 (1), 159-174.
22. Lesaigle, E.M. and Biers, D.W., 2000. Effect of type of information on real time usability evaluation: implications for remote usability testing. In: Proceedings of the XIVth triennial congress of the international ergonomics association and 44th annual meeting of the human factors and ergonomics society, 29 July-4 August, San Diego, CA. Santa Monica, CA: HFES, 6-585-6-588.
23. Lewis, C., 1982. Using the 'thinking-aloud' method in cognitive interface design. Research report RC9265. Yorktown Heights, NY: IBM Watson Research Center.
24. Meissner, CA. and Brigham, J.C., 2001. A meta-analysis of the verbal overshadowing effect in face identification. Applied Cognitive Psychology, 15 (6), 603-616.
25. Monk, A., Wright, P., Haber, J., and Davenport, L., 1993. Improving your human-computer interface: a practical technique. New York: Prentice Hall.
26. Nielsen, J., 1993. Usability engineering. Boston: Academic Press.
27. Nisbett, R.E. and Wilson, T.D., 1977. Telling more than we can know: verbal reports on mental processes. Psychological Review, 84 (3), 231-259.
28. Nygren, T.E., 1991. Psychometric properties of subjective workload measurement techniques: implications for their use in the assessment of perceived mental workload. Human Factors, 33 (1), 17-33.
29. Norgaard, M. and Hornbask, K., 2006. What do usability evaluators do in practice? An explorative study of think-aloud testing. In: Proceedings of the sixth DIS conference on designing interactive systems, 26-28 June, University Park, PA. New York: ACM Press, 209-218.
30. Rayner, K., 1998. Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124 (3), 372-422.
31. Rhenius, D. and Deffner, G., 1990. Evaluation of concurrent thinking aloud using eye-tracking data. In: Proceedings of the human factors society 34th annual meeting, 8-12 October, Orlando, FL, Santa Monica, CA: HFS, 1265-1269.
32. Russo, J.E., Johnson, E.J., and Stephens, D.L., 1989. The validity of verbal protocols. Memory and Cognition, 17 (6), 759-769.
33. Salvucci, D.D. and Goldberg, J.H., 2000. Identifying fixations and saccades in eye-tracking protocols. In: Proceedings of the eye tracking research and applications symposium 2000, 6-8 November, Palm Beach Gardens, FL. New York: ACM Press, 71-78.
34. Schooler, J.W., 2002. Verbalization produces a transfer inappropriate processing shift. Applied Cognitive Psychology, 16(8), 989-997.
35. Schooler, J.W. and Engstler-Schooler, T.Y., 1990. Verbal overshadowing of visual memories: some things are better left unsaid. Cognitive Psychology, 22 (1), 36-71.
36. Spool, J., Scanlon, T., Schroeder, W., Snyder, C., and DeAngelo, T., 1999. Web site usability: a designer's guide. San Francisco, CA: Morgan Kaufmann.
37. Van den Haak, M.J., de Jong, M.D.T., and Schellens, P.J., 2003. Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behaviour & Information Technology. 22 (5), 339-351.
38. Van den Haak, M.J., de Jong, M.D.T., and Schellens, P.J., 2004. Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison. Interacting with Computers, 16(6), 1153-1170.
39. Wilson, T.D. and Schooler, J.W., 1991. Thinking too much: introspection can reduce the quality of preferences and decisions. Journal of Personality and Social Psychology, 60(2), 181-192.
40. Wixon, D., 2003. Evaluating usability methods: why the current literature fails the practitioner. ACM Interactions, 10 (4), 29-34.
41. Wood, R.E., 1986. Task complexity: definition of the construct. Organizational Behavior and Human Decision Processes, 37 (1), 60-82.
42. Wright, R.B. and Converse, S.A., 1992. Method bias and concurrent verbal protocol in software usability testing. In: Proceedings of the human factors society 36th annual meeting, 12-16 October, Atlanta, GA. Santa Monica, CA: HFS, 1220-1224.