Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue

Behaviour and Information Technology

Volumn 22, Issue 5, 2003, Pages 339-351

  Van Den Haak, Maaike J ^a   De Jong, Menno D T ^a   Schellens, Peter Jan ^a  

^a UNIVERSITY OF TWENTE   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL COMPLEXITY; COMPUTER SOFTWARE REUSABILITY; COMPUTER SOFTWARE SELECTION AND EVALUATION; DATA REDUCTION; ONLINE SYSTEMS;

USABILITY TESTING;

NETWORK PROTOCOLS;

EID: 0242270033     PISSN: 0144929X     EISSN: None     Source Type: Journal    
DOI: 10.1080/0044929031000     Document Type: Article

References (36)

1. ALLWOOD, C. M. and KALÉN, T. 1997, Evaluating and improving the usability of a user manual. Behaviour & Information Technology, 16, 43-57.
2. BARNUM, C. M. 2002, Usability Testing and Research (New York: Longman).
3. BATTLESON, B., BOOTH, A. and WEINTROP, J. 2001, Usability testing of an academic library web site: a case study. Journal of Academic Librarianship, 237, 188-198.
4. BOLTON, R. L. 1993, Pretesting questionnaires: content analyses of respondents' concurrent verbal protocols. Marketing Science, 12, 280-303.
5. BOREN, M. T. and RAMEY, J. 2000, Thinking aloud: reconciling theory and practice. IEEE Transactions on Professional Communication, 43, 261-278.
6. BOWERS, V. A. and SNYDER, H. L. 1990, Concurrent versus retrospective verbal protocols for comparing window usability. Human Factors Society 34th Meeting, 8-12 October 1990 (Santa Monica: HFES), pp. 1270-1274.
7. BRANCH, J. L. 2000, Investigating the information-seeking processes of adolescents: The value of using think alouds and think afters. Library & Information Science Research, 22, 371-392.
8. CAMPBELL, N. (ed.) 2001, Usability Assessment of Library-Related Web Sites: Methods and Case Studies. (Chicago: LITA).
9. CAULTON, D. A. 2001, Relaxing the homogeneity assumption in usability testing. Behaviour & Information Technology, 20, 1-7.
10. DE JONG, M. and SCHELLENS, P. J. 2000, Toward a document evaluation methodology: what does research tell us about the validity and reliability of methods? IEEE Transactions on Professional Communication, 43, 242-260.
11. DIELI, M. 1986, Designing Successful Documents: An Investigation of Document Evaluation Methods (Dissertation Carnegie Mellon University, Pittsburgh, PA).
12. DUMAS, J. S. and REDISH, J. C. 1999, A Practical Guide to Usability Testing, Revised edition (Exeter: Intellect).
13. ERICSSON, K. A. and SIMON, H. A. 1993, Protocol Analysis: Verbal Reports as Data (Cambridge, MA: MIT Press).
14. GRAY, W. D. and SALZMAN, M. C. 1998, Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13, 203-261.
15. HALL, M., DE JONG, M. and STEEHOUDER, M. forthcoming, Cultural differences and usability evaluation; individualistic and collectivistic participants compared.
16. HASSENZAHL, M. 2000, Prioritizing usability problems: data-driven and judgement-driven severity estimates. Behaviour & Information Technology, 19, 29-42.
17. HENDERSON, R. D., SMITH, M. C., PODD, J. and VARELA-ALVAREZ, H. 1995, A comparison of the four prominent user-based methods for evaluating the usability of computer software. Ergonomics, 38, 2030-2044.
18. HOC, J. M. and LEPLAT, J. 1983, Evaluation of different modalities of verbalization in a sorting task. International Journal of Man-Machine Studies, 18, 283-306.
19. JANSEN, C. and STEEHOUDER, M. 1992, Forms as a source of communication problems. Journal of Technical Writing and Communication, 22, 179-194.
20. JOHN, B. E. and MARKS, S. J. 1997, Tracking the effectiveness of usability evaluation methods. Behaviour & Information Technology, 16, 188-202.
21. KUUSELA, H. and PAUL, P. 2000, A comparison of concurrent and retrospective verbal protocol analysis. American Journal of Psychology, 113, 387-404.
22. LAVERY, D., COCKTON, G. and ATKINSON, M.P. 1997, Comparison of evaluation methods using structured usability problem reports. Behaviour & Information Technology, 16, 246-266.
23. LEWIS, J.R. 1994, Sample sizes for usability studies: additional considerations. Human Factors, 36, 369-378.
24. LOXTERMAN, J. A., BECK, I. L. and MCKEOWN, M. G. 1994, The effects of thinking aloud during reading on students' comprehension of more or less coherent text. Reading Research Quarterly, 29, 353-367.
25. NIELSEN, J. 1993, Usability Engineering (Boston, MA: Academic Press).
26. NIELSEN, J. 1994, Estimating the number of subjects needed for a thinking aloud test. International Journal of Human-Computer Studies, 41, 385-397.
27. NORLIN, E. and WINTERS, C. M. I. 2002, Usability testing for library websites: a hands-on guide (Chicago: American Library Association).
28. RUBIN, J. 1994, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests (New York: Wiley).
29. RUSSO, J. E., JOHNSON, E. J. and STEPHENS, D. L. 1989, The validity of verbal protocols. Memory & Cognition, 17, 759-769.
30. SCHRIVER, K.A. 1997, Dynamics in Document Design; Creating Text for Readers (New York: Wiley).
31. SIENOT, M. 1997, Pretesting web sites; a comparison between the plus-minus method and the think-aloud method for the World Wide Web. Journal of Business and Technical Communication, 11, 469-482.
32. SMILOWITZ, E. D., DARKELL, M. J. and BENSON, A. E. 1994, Are we overlooking some usability testing methods? A comparison of lab, beta, and forum tests. Behaviour & Information Technology, 13, 183-190.
33. TAYLOR, K. L. and DIONNE, J. P. 2000, Accessing problem-solving strategy knowledge: The complementary use of concurrent verbal protocols and retrospective debriefing. Journal of Educational Psychology, 29, 413-425.
34. TEAGUE, R., DE JESUS, K. and NUNES-UENO, M. 2001, Concurrent vs. post-task usability test ratings. Conference on Human Factors and Computing Systems, 31 March - 5 April 2001 (Seattle, WA: ACM SIGCHI), pp. 289-290.
35. VIRZI, R. A. 1992, Refining the test phase of usability evaluation: how many subjects is enough? Human Factors, 34, 457-468.
36. WRIGHT, P. C. and MONK, A. F. 1991, A cost-effective evaluation method for use by designers. International Journal of Man-Machine Studies, 36, 544-565.