Combining concurrent think-aloud protocols and eye-tracking observations: An analysis of verbalizations and silences

IEEE Transactions on Professional Communication

Volumn 55, Issue 3, 2012, Pages 206-220

  Elling, Sanne ^a   Lentz, Leo ^a   De Jong, Menno ^b  

^a UTRECHT UNIVERSITY   (Netherlands)

^b UNIVERSITY OF TWENTE   (Netherlands)

Author keywords

Concurrent think aloud (CTA) protocols; eye tracking; usability; website evaluation

Indexed keywords

EYE MOVEMENTS; USABILITY ENGINEERING; WEB DESIGN;

CONCURRENT THINK-ALOUD PROTOCOL; EYE TRACKING TECHNOLOGIES; EYE-TRACKING; RESEARCH PROBLEMS; THINK ALOUD; THINK-ALOUD PROTOCOL; USABILITY; USABILITY TESTING; WEBSITE EVALUATION; WORKING PROCESS;

EYE TRACKING;

EID: 84865494121     PISSN: 03611434     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPC.2012.2206190     Document Type: Article

References (41)

1. R. E. Nisbett and T. D. Wilson, "Telling more than we can know: Verbal reports on mental processes", Psychol. Rev., vol. 84, no. 3, pp. 231-259, 1977.
2. K. A. Ericsson and H. A. Simon, Protocol Analysis: Verbal Reports as Data (Rev. ed.). Cambridge, MA: MIT Press, 1993.
3. T. Boren and J. Ramey, "Thinking aloud: Reconciling theory and practice", IEEE Trans. Prof. Commun., vol. 43, no. 3, pp. 261-278, Sep. 2000.
4. E. Krahmer and N. Ummelen, "Thinking about thinking aloud: A comparison of two verbal protocols for usability testing", IEEE Trans. Prof. Commun., vol. 47, no. 2, pp. 105-117, Jun. 2004.
5. M. Nørgaard and K. Hornbæk, "What do usability evaluators do in practice? An explorative study of think-aloud testing", in Proc. 6th Conf. Design. Interact. Syst. ACM, 2006, pp. 209-218. (Pubitemid 44725733)
6. J. Ramey, T. Boren, E. Cuddihy, J. Dumas, Z. Guan, M. J. Van den Haak, and M. D. T. De Jong, "Does think aloud work? How do we know?", in Proc. CHI Extended Abstracts Human Factors Comput. Syst., 2006, pp. 45-48.
7. M. Hertzum, K. D. Hansen, and H. H. K. Andersen, "Scrutinising usability evaluation: Does thinking aloud affect behaviour and mental workload?", Behav. Inf. Technol., vol. 28, no. 2, pp. 165-181, 2009.
8. E. L. Olmsted-Hawala, E. D. Murphy, S. Hawala, and K. T. Ashenfelter, "Think-aloud protocols: A comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability", in Proc. 28th Int. Conf. Human Factors Comput. Syst., 2010, pp. 2381-2390.
9. T. Zhao and S. McDonald, "Keep talking: An analysis of participant utterances gathered using two concurrent think-aloud methods", in Proc. NordiCHI, 2010, pp. 581-590.
10. V. A. Bowers and H. L. Snyder, "Concurrent versus retrospective verbal protocol for comparing Windows usability", in Proc. 34th Human Factors Soc. Annu. Meeting, 1990, pp. 1270-1274.
11. N. Ummelen and R. Neutelings, "Measuring reading behavior in policy documents: A comparison of two instruments", IEEE Trans. Prof. Commun., vol. 43, no. 3, pp. 292-301, Sep. 2002.
12. M. J. Van den Haak, M. D. T. De Jong, and P. J. Schellens, "Retrospective versus concurrent think-aloud protocols: Testing the usability of an online library catalogue", Behav. Inf. Technol., vol. 22, no. 5, pp. 339-351, 2003.
13. M. J. Van den Haak, M. D. T. De Jong, and P. J. Schellens, "Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: A methodological comparison", Interact. With Comput., vol. 16, no. 6, pp. 1153-1170, 2004.
14. M. J. Van den Haak, M. D. T. De Jong, and P. J. Schellens, "Constructive interaction: An analysis of verbal interaction in a usability setting", IEEE Trans. Prof. Commun., vol. 49, no. 4, pp. 311-324, Dec. 2006. (Pubitemid 46405741)
15. M. J. Van den Haak, M. D. T. De Jong, and P. J. Schellens, "Evaluation of an informational Web site: Three variants of the think-aloud method compared", Tech. Commun., vol. 54, no. 1, pp. 58-71, 2007.
16. M. J. Van den Haak, M. D. T. De Jong, and P. J. Schellens, "Evaluating municipal websites: A methodological comparison of three think-aloud variants", Government Inf. Quart., vol. 26, no. 1, pp. 193-202, 2009.
17. S. Elling, L. Lentz, and M. De Jong, "Retrospective think-aloud method: Using eye movements as an extra cue for participants' verbalizations", in Proc. 29th Int. Conf. Human Factors Comput. Syst. ACM, 2011, pp. 1161-1170.
18. L. Van Waes, "Thinking aloud as a method for testing the usability of websites: The influence of task variation on the evaluation of hypertext", IEEE Trans. Prof. Commun., vol. 43, no. 3, pp. 279-291, Sep. 2002.
19. L. Cooke, "Eye tracking: How it works and how it relates to usability", Tech. Commun., vol. 52, no. 4, pp. 456-463, 2005. (Pubitemid 41687568)
20. L. Cooke and E. Cuddihy, "Using eye tracking to address limitations in think-aloud protocol", in Proc. IEEE Int. Prof. Commun. Conf, 2005, pp. 653-658.
21. T. Van Gog, F. Paas, J. J. G. Van Merrienboer, and P. Witte, "Uncovering the problem-solving process: Cued retrospective reporting versus concurrent and retrospective reporting", J. Exp. Psychol: Appl, vol. 11, no. 4, pp. 237-244, 2005.
22. L. J. Ball, N. Eger, R. Stevens, and J. Dodd, "Applying the PEEP method in usability testing", Interfaces, vol. 67, pp. 15-19, 2006.
23. Z. Guan, S. Lee, E. Cuddihy, and J. Ramey, "The validity of the stimulated retrospective think-aloud method as measured by eye tracking", in Proc. SIGCHI Conf. Human Factors Comput. Syst ACM, 2006, pp. 1253-1262. (Pubitemid 44032224)
24. N. Eger, L. J. Ball, R. Stevens, and J. Dodd, "Cueing retrospective verbal reports in usability testing through eye-movement replay", in Proc. 21st Brit. CHI Group Annu. Conf. HCI: People Comput. XXI: HCI... But Not as We Know It, 2007, vol. 1, pp. 129-137.
25. C. Ehmke and S. Wilson, "Identifying web usability problems from eye-tracking data", in Proc. 21st British CHI Group Annu. Conf. HCI: People Comput. XXI... But Not as We Know It, 2007, vol. 1, pp. 119-128.
26. A. Hyrskykari, S. Ovaska, K. Räihä, P. Majaranta, and M. Lehtinen, "Gaze path stimulation in retrospective think-aloud", J. Eye Movement Res., vol. 2, no. 4, pp. 1-18, 2008.
27. T. Van Gog, L. Kester, F. Nievelstein, B. Giesbers, and F. Paas, "Uncovering cognitive processes: Different techniques that can contribute to cognitive load research and instruction", Comput. Human Behav., vol. 25, no. 2, pp. 325-331, 2009.
28. L. Cooke, "Assessing concurrent think-aloud protocol as a usability test method: A technical communication approach", IEEE Trans. Prof. Commun., vol. 35, no. 3, pp. 202-215, Sep. 2010.
29. M. A. Just and P. A. Carpenter, "Eye fixations and cognitive processes", Cognit. Psychol., vol. 8, no. 4, pp. 441-480, 1976.
30. K. Rayner, "Eye movements in reading and information processing: 20 years of research", Psychol. Bull., vol. 124, no. 3, pp. 372-422, 1998. (Pubitemid 128461792)
31. J. H. Goldberg and A. M. Wichansky, "Eye tracking in usability evaluation: A practitioner's guide", in The Mind's Eye: Cognitive and Applied Aspects of Eye Movement Research, J. Hyönä, R. Radach, and H. Deubel, Eds. New York: Elsevier, 2003, pp. 493-516.
32. R. J. K. Jacob and K. S. Karn, "Eye tracking in human-computer interaction and usability research: Ready to deliver the promises", in The Mind's Eye: Cognitive and Applied Aspects of Eye Movement Research, J. Hyönä, R. Radach, and H. Deubel, Eds. New York: Elsevier, 2003, pp. 574-605.
33. S. Monsell, "Task switching", Trends Cognit. Sci., vol. 7, no. 3, pp. 134-140, 2003. (Pubitemid 36294174)
34. M. A. Neerincx, J. Lindenberg, and S. Pemberton, "Support concepts for web navigation: A cognitive engineering approach", in Proc. 10th Int. Conf. World Wide Web, Hong Kong, China, 2001, pp. 119-128.
35. S. Elling, L. Lentz, and M. De Jong, "Users' abilities to review website pages", J. Bus. Tech. Commun., vol. 26, no. 2, pp. 170-200, 2012.
36. S. Elling, L. Lentz, M. De Jong, and H. Van den Bergh, "Measuring the quality of governmental websites in a controlled versus an online setting with the "Website Evaluation Questionnaire", Gov. Inf. Quart., vol. 29, no. 3, pp. 383-393, 2012.
37. W. P. Eveland and S. Dunwoody, "Examining information processing on the World Wide Web using think aloud protocols", Media Psychol, vol. 2, no. 3, pp. 219-244, 2004.
38. A. L. Cox and M. M. Silva, "The role of mouse movements in interactive search", in Proc. 28th Annu. Meeting Cognit. Sci. Soc., Vancouver, BC, Canada, 2006, pp. 1156-1161.
39. J. H. Goldberg and X. P. Kotval, "Computer interface evaluation using eye movements: Methods and constructs", Int. J. Ind. Ergonom., vol. 24, no. 6, pp. 631-645, 1999. (Pubitemid 29447641)
40. L. Cowen, L. J. Ball, and J. Delin, "An eye-movement analysis of web-page usability", in People Comput. XVI (Proc. HCI 2002). London, UK: Springer, 2002, pp. 317-335.
41. A. Poole, L. J. Ball, and P. Philips, "In search of salience: A response time and eye movement analysis of bookmark recognition", in People Comput. XVIII (Proc. HCI 2004). London, UK: Springer, 2004, pp. 363-378.