The evaluator effect: A chilling fact about usability evaluation methods

International Journal of Human-Computer Interaction

Volumn 15, Issue 1, 2003, Pages 183-204

  Hertzum, Morten ^a   Jacobsen, Niels Ebbe ^b  

^a TECHNICAL UNIVERSITY OF DENMARK   (Denmark)

^b NONE

Author keywords

[No Author keywords available]

Indexed keywords

COGNITIVE SYSTEMS; INTERFACES (COMPUTER); LARGE SCALE SYSTEMS; PROBLEM SOLVING;

USABILITY EVALUATION METHODS;

USABILITY ENGINEERING;

EID: 0037243757     PISSN: 10447318     EISSN: None     Source Type: Journal    
DOI: 10.1207/S15327590IJHC1501_14     Document Type: Conference Paper

References (34)

1. Connell, I. W., & Hammond, N. V. (1999). Comparing usability evaluation principles with heuristics: Problem instances vs. problem types. In M. Angela Sasse & C. Johnson (Eds.), Proceedings of the IFIP INTERACT '99 Conference on Human-Computer Interaction (pp. 621-629). Amsterdam: IOS.
2. Corona, R., Mele, A., Amini, M., De Rosa, G., Coppola, G., Piccardi, P., Fucci, M., Pasquini, P., & Faraggiana, T. (1996). Interobserver variability on the histopathologic diagnosis of cutaneous melanoma and other pigmented skin lesions. Journal of Clinical Oncology, 14, 1218-1223.
3. Cramer, S. F. (1997). Interobserver variability in dermatopathology. Archives of Dermatology, 133, 1033-1036.
4. Dumas, J. S., & Redish, J. C. (1993). A practical guide to usability testing. Norwood, NJ: Ablex.
5. Dutt, A., Johnson, H., & Johnson, P. (1994). Evaluating evaluation methods. In G. Cockton, S. W. Draper, & G. R. S. Weir (Eds.), People and computers IX (pp. 109-121). Cambridge, England: Cambridge University Press.
6. Funk, M. E., Reid, C. A., & McCoogan, L. S. (1983). Indexing consistency in MEDLINE. Bulletin of the Medical Library Association, 71(2), 176-183.
7. Gray, W. D., & Salzman, M. C. (1998). Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13, 203-261.
8. Hertzum, M., & Jacobsen, N. E. (1999). The evaluator effect during first-time use of the cognitive walkthrough technique. In H.-J. Bullinger & J. Ziegler (Eds.), Human-computer interaction: Ergonomics and user interfaces (Vol. 1, pp. 1063-1067). London: Lawrence Erlbaum Associates, Inc.
9. Jacobsen, N. E., Hertzum, M., & John, B. E. (1998). The evaluator effect in usability studies: Problem detection and severity judgments. In Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting (pp. 1336-1340). Santa Monica, CA: Human Factors and Ergonomics Society.
10. Jacobsen, N. E., & John, B. E. (2000). Two case studies in using cognitive walkthrough for interface evaluation (CMU Technical Report No. CMU-CS-00-132). Pittsburgh, PA: Carnegie Mellon University.
11. Lesaigle, E. M., & Biers, D. W. (2000). Effect of type of information on real time usability evaluation: Implications for remote usability testing. In Proceedings of the XIVth Triennial Congress of the International Ergonomics Association and 44th Annual Meeting of the Human Factors and Ergonomics Society (pp. 6-585-6-588). Santa Monica, CA: Human Factors and Ergonomics Society.
12. Lewis, C. (1982). Using the "thinking-aloud" method in cognitive interface design (IBM Research Rep. No. RC 9265 [#40713]). Yorktown Heights, NY: IBM Thomas J. Watson Research Center.
13. Lewis, C., Polson, P., Wharton, C., & Rieman, J. (1990). Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces. In Proceedings of the ACM CHI '90 Conference (pp. 235-242). New York: ACM.
14. Lewis, C., & Wharton, C. (1997). Cognitive walkthroughs. In M. Helander, T. K. Landauer, & P. Prabhu (Eds.), Handbook of human-computer interaction (Rev. 2nd ed., pp. 717-732). Amsterdam: Elsevier.
15. Lewis, J. R. (1994). Sample sizes for usability studies: Additional considerations. Human Factors, 36, 368-378.
16. Miller, J., Wood, M., & Roper, M. (1998). Further experiences with scenarios and checklists. Empirical Software Engineering, 3(1), 37-64.
17. Molich, R., Bevan, N., Curson, I., Butler, S., Kindlund, E., Miller, D., & Kirakowski, J. (1998). Comparative evaluation of usability tests. In Proceedings of the Usability Professionals Association 1998 Conference (pp. 189-200). Chicago: UPA.
18. Molich, R., Thomsen, A. D., Karyukina, B., Schmidt, L., Ede, M., van Oel, W., & Arcuri, M. (1999). Comparative evaluation of usability tests. In Extended Abstracts of ACM CHI '99 Conference (pp. 83-84). New York: ACM.
19. Nielsen, J. (1992). Finding usability problems through heuristic evaluation. In Proceedings of the ACM CHI '92 Conference (pp. 373-380). New York: ACM.
20. Nielsen, J. (1993). Usability engineering. Boston: Academic.
21. Nielsen, J. (1994a). Heuristic evaluation. In J. Nielsen & R. L. Mack (Eds.), Usability inspection methods (pp. 25-62). New York: Wiley.
22. Nielsen, J. (Ed.). (1994b). Usability laboratories [Special issue]. Behaviour & Information Technology, 13(1 & 2).
23. Nielsen, J., & Landauer, T. K. (1993). A mathematical model of the finding of usability problems. In Proceedings of the INTERCHI '93 Conference (pp. 206-213). New York: ACM.
24. Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. In Proceedings of the ACM CHI '90 Conference (pp. 249-256). New York: ACM.
25. Olson, G. M., & Moran, T. P. (Eds.). (1998). Commentary on "Damaged merchandise?" Human-Computer Interaction, 13, 263-323.
26. Polson, P., & Lewis, C. (1990). Theory-based design for easily learned interfaces. Human-Computer Interaction, 5, 191-220.
27. Polson, P., Lewis, C., Rieman, J., & Wharton, C. (1992). Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. International Journal of Man-Machine Studies, 36, 741-773.
28. Rowley, D. E., & Rhoades, D. G. (1992). The cognitive jogthrough: A fast-paced user interface evaluation procedure. In Proceedings of the ACM CHI '92 Conference (pp. 389-395). New York: ACM.
29. Sievert, M. C., & Andrews, M. J. (1991). Indexing consistency in Information Science Abstracts. Journal of the American Society for Information Science, 42(1), 1-6.
30. Sørensen, J. B., Hirsch, F. R., Gazdar, A., & Olsen, J. E. (1993). Interobserver variability in histopathologic subtyping and grading of pulmonary adenocarcinoma. Cancer, 71, 2971-2976.
31. Virzi, R. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34, 457-468.
32. Wharton, C., Bradford, J., Jeffries, R., & Franzke, M. (1992). Applying cognitive walkthroughs to more complex user interfaces: Experiences, issues, and recommendations. In Proceedings of the ACM CHI '92 Conference (pp. 381-388). New York: ACM.
33. Wharton, C., Rieman, J., Lewis, C., & Poison, P. (1994). The cognitive walkthrough method: A practitioner's guide. In J. Nielsen & R. L. Mack (Eds.), Usability inspection methods (pp. 105-140). New York: Wiley.
34. Zunde, P., & Dexter, M. E. (1969). Indexing consistency and quality. American Documentation, 20, 259-267.