Dogmas in the assessment of usability evaluation methods

Behaviour and Information Technology

Volumn 29, Issue 1, 2010, Pages 97-111

  Hornbk, Kasper ^a  

^a UNIVERSITY OF COPENHAGEN   (Denmark)

Author keywords

Downstream utility; Formative evaluation; Methodology; Usability evaluation methods; Usability testing

Indexed keywords

EVALUATION PROCESS; FORMATIVE EVALUATION; INADEQUATE PROCEDURES; REAL-WORLD SYSTEM; RESEARCH APPROACH; SYSTEMS DEVELOPMENT; USABILITY EVALUATION METHODS; USABILITY PROBLEMS; USABILITY TESTING;

USABILITY ENGINEERING;

EID: 77649298100     PISSN: 0144929X     EISSN: 13623001     Source Type: Journal    
DOI: 10.1080/01449290801939400     Document Type: Article

References (66)

1. Andre, T.S., et al., 2001. The user action framework: a reliable foundation for usability engineering support tools. International Journal of Human-Computer Studies, 54 (1), 107-136.
2. Bastian, J.M.C., Scapin, D.L., and Leulier, C., 1999. The ergonomic criteria and the ISO/DIS 9241-10 dialogue principles: a pilot comparison in an evaluation task. Interacting with Computers, 11, 299-322.
3. Blandford, A., et al., 2004. Scoping analytical usability evaluation techniques: a case study. CASSM Working paper, available from http://www.uclic.ucl.ac.uk/annb/ CASSM/CASSMpubls.html
4. Boren, T. and Ramey, J., 2000. Thinking aloud: reconciling theory and practice. IEEE Transactions on Professional Communication, 43 (3), 261-277.
5. Borgholm, T. and Madsen, K.H., 1999. Cooperative usability practices. Communications of the ACM, 42 (5), 91-97.
6. Chi,M.T.H., 1997. Quantifying qualitative analyses of verbal data: a practical guide. The Journal of the Learning Sciences, 6 (3), 271-315.
7. Cockton, G., Lavery, D., and Woolrych, A., 2003a. Inspection-based evaluations. In: J.A. Jacko and A. Sears, eds. TheHuman-Computer Interaction Handbook. Mahwah, NJ: Lawrence Erlbaum Associates, 1118-1138.
8. Cockton, G. and Woolrych, A., 2001. Understanding inspection methods: lessons from an assessment of heuristic evaluation. In: A. Blandford, J. Vanderdonckt, and P.D. Gray, eds. Proceedings of people and computers XV: joint proceedings of HCI 2001 and IHM 2001. Berlin: Springer-Verlag, 171-192.
9. Cockton, G. and Woolrych, A., 2002. Sale must end: should discount methods be cleared off HCI's shelves? Interactions, 9 (5), 13-18.
10. Cockton, G., et al., 2003b. Changing analysts' tunes: the surprising impact of a new instrument for usability inspection method assessment. In: Proceedings of people and computers XVII: designing for society. Springer Verlag, 145-162.
11. Cockton, G., Woolrych, A., and Hindmarch, M., 2004. Reconditioned merchandise: extended structured report formats in usability inspection. In: Extended abstracts of ACM Conference on Human-Computer Interaction. New York: ACM Press, 1433-1436.
12. Connell, I., Blandford, A., and Green, T., 2004. CASSM and cognitive walkthrough: usability issues with ticket wending machines. Behaviour & Information Technology, 23 (5), 307-320.
13. Connell, I.W. and Hammond, N.V., 1999. Comparing usability evaluation principles with heuristics: problem instances vs. problem types. In: Proceedings of IFIP TC.13 international conference on human-computer interaction, Amsterdam: IOS Press, 621-629.
14. de Angelli, A., et al., 2003. On the advantages of a systematic inspection for evaluating hypermedia usability. International Journal of Human-Computer Interaction, 15 (3), 315-335.
15. De Jong, M. and Schellens, P.J., 2000. Towards a document evaluation methodology: what does research tell us about the validity and reliability of evaluation methods? IEEE Transactions on Professional Communication, 43 (3), 242-260.
16. Desurvire, H.W., Kondziela, J.M., and Atwood, M.E., 1992. What is gained and lost when using evaluation methods other than empirical testing. In: Proceedings of people and computers VII, Cambridge University Press, 89-102.
17. Doubleday, A., Ryan, A., and Sutcliffe, A., 1997. A comparison of usability techniques for evaluating design. In: Proceedings of ACM conference on designing interactive systems. New York: ACM Press, 101-110.
18. Frøkjær, E. and Hornbæk, K., 2002. Metaphors of human thinking in HCI: habit, stream of thought, awareness, utterance, and knowing. In: Proceedings of HF2002/ OzCHI 2002.
19. Frøkjær, E. and Larusdottir, M., 1999. Predicting of usability: comparing method combinations. In: Proceedings of 10th international conference of the information resources management association.
20. Fu, L. and Salvendy, G., 2002. Effectiveness of user-testing and heuristic evaluation as a function of performance classification. Behaviour & Information Technology, 21 (2), 137-143.
21. Gray, W.D. and Salzman, M.C., 1998. Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13 (3), 203-261.
22. Grudin, J., 1991. Interactive systems: bridging the gaps between developers and users. IEEE Computer, 24 (4), 59-69.
23. Gulliksen, J., et al., 2004. Making a difference - a survey of the usability profession in Sweden. In: Proceedings of proceedings of the third nordic conference on humancomputer interaction. New York: ACM Press, 207-215.
24. Hartson, H.R., Andre, T.S., and Williges, R.C., 2001. Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 13 (4), 373-410.
25. Hassenzahl, M., 2000. Prioritising usability problems: datadriven and judgement-driven severity estimates. Behaviour & Information Technology, 19, 29-42.
26. Helms Jørgensen, A., 1990. Thinking-aloud in user interface design: a method promoting cognitive ergonomics. Ergonomics, 33 (4), 501-507.
27. Hertzum, M. and Jacobsen, N.E., 2001. The evaluator effect: a chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 13, 421-443.
28. Hertzum, M., Jacobsen, N.E., and Molich, R., 2002. Usability inspections by groups of specialist: perceived agreement in spite of disparate observations. In: Proceedings of extended abstracts of the ACM conference on human factors in computing, New York: ACM Press, 662-663.
29. Hornbæk, K. and Frøkjær, E., 2004a. Usability inspection by metaphors of human thinking compared to heuristic evaluation. International Journal of Human-Computer Interaction, 17 (3), 357-374.
30. Hornbæk, K. and Frøkjær, E., 2004b. Two psychology-based usability inspection techniques studied in a diary experiment. In: 3rd Nordic conference on human-computer interaction (Nordichi 2004), New York: ACMPress, 3-12.
31. Hornbæk, K. and Frøkjær, E., 2005. Comparing usability problems and redesign proposals as input to practical systems development. In: Proceedings of ACM conference on human factors in computing systems. New York: ACM Press, 391-400.
32. Huart, J., Kolski, C., and Sagar, M., 2004. Evaluation of multimedia applications using inspection methods: the cognitive walkthrough case. Interacting with Computers, 16, 183-215.
33. ISO, 1998. Ergonomic requirements for office work with visual display terminals (VDTs)-part 11: guidance on usability.
34. Jacobsen, N.E. and John, B.E., 2000. Two case studies in using cognitive walkthroughs for interface evaluation (Carnegie Mellon Tech Rep. No. CMU-CS-00-132).
35. Jeffries, R., 1994. Usability problem reports: helping evaluators communicate effectively with developers. In: J. Nielsen and R.L. Mack, eds. Usability Inspection Methods. New York: Wiley, 273-294.
36. Jeffries, R., et al., 1991. User interface evaluation in the real world: a comparison of four techniques. In: Proceedings of ACM conference on human factors in computing, New York: ACM Press, 119-124.
37. John, B., et al., 2004. Predictive human performance modelling made easy. In: Proceedings of ACM conference on human factors in computing systems, New York: ACM Press, 455-462.
38. John, B. and Mashyna, M.M., 1995. Evaluating a multimedia authoring tool with cognitive walkthrough and think-aloud user studies (Rep. No. CMU-HCII-95-105/ CMU-CS-95-189).
39. John, B.E. and Mashyna, M.M., 1997. Evaluating a multimedia authoring tool. Journal of the American Society of Information Science, 48 (9), 1004-1022.
40. John, B.E. and Marks, S.J., 1997. Tracking the effectiveness of usability evaluation methods. Behaviour & Information Technology, 16 (4/5), 188-202.
41. Karat, C.-M., Campbell, R., and Fiegel, T., 1992. Comparison of empirical testing and walkthrough methods in usability interface evaluation. In: Proceedings of ACM conference on human factors in computing. New York: ACM Press, 397-404.
42. Keenan, S.L., et al., 1999. The usability problem taxonomy: a framework for classification and analysis. Empirical Software Engineering, 4 (1), 71-104.
43. Kjeldskov, J. and Skov, M., 2003. Creating realistic laboratory settings: comparative studies of three thinkaloud usability evaluations of a mobile system. In: Proceedings of 9th IFIP TC13 international conference on human-computer interaction. Amsterdam: IOS Press, 663-670.
44. Kjeldskov, J. and Stage, J., 2004. New techniques for usability evaluation of mobile systems. International Journal of Human-Computer Studies, 60, 599-620.
45. Klein, G., 1999. Sources of power: how people make decisions. Cambridge, MA: MIT Press.
46. Kukla, A., 1998. Studies in scientific realism. Oxford University Press.
47. Lavery, D., Cockton, G., and Atkinson, M.P., 1997. Comparison of evaluation methods using structured usability problem reports. Behaviour & Information Technology, 16 (4/5), 246-266.
48. Law, E. and Hvannberg, E., 2004. Analysis of combinatorial user effect in international usability tests. In: Proceedings of ACM conference on human factors in computing systems, New York: ACM Press, 9-16.
49. Mack, R.L. and Montaniz, F., 1994. Observing, predicting, and analyzing usability problems. In: J. Nielsen and R.L. Mack, eds. Usability Inspection Methods. Wiley, 295-339.
50. Mankoff, J., et al., 2003. Heuristic evaluation of ambient displays. CHI letters, CHI 2003, ACM conference on human factors in computing systems, 5 (1), 169-176.
51. Markopoulos, P. and Bekker, M., 2003. On the assessment of usability testing methods for children. Interacting with Computers, 15, 227-243.
52. Molich, R., et al., 2004. Comparitive usability evaluation. Behaviour & Information Technology, 23 (1), 65-74.
53. Nielsen, J., 1992. Finding usability problems through heuristic evaluation. In: P. Baursfield, J. Bennet, and G. Lynch, eds. Proceedings of ACM CHI'92 conference on human factors in computing systems. New York: ACM Press, 373-380.
54. Nielsen, J. and Phillips, V., 1993. Estimating the relative usability of two interfaces: heuristic, formal, and empirical methods compared. In: Proceedings of ACM conference in human factors in computing systems, 214-221.
55. Nørgaard, M. and Hornbæk, K., 2006. What do usability evaluators do in practice? An explorative study of thinkaloud testing. In: ACM symposium on designing interactive systems, DIS2006. New York: ACM Press, 209-218.
56. Sawyer, P., Flanders, A., and Wixon, D., 1996. Making a difference - the impact of inspections. In: M.J. Tauber, ed. Proceedings of ACM conference on human factors in computing. New York: ACM Press, 376-382.
57. Sears, A. and Hess, D., 1999. Cognitive walkthroughs: understanding the effect of task description detail on evaluator performance. International Journal of Human-Computer Interaction, 11, 185-200.
58. Smith, A. and Dunckley, L., 2002. Prototype evaluation and redesign: structuring the design space through contextual techniques. Interacting with Computers, 14, 821-843.
59. Spool, J. and Shroeder, W., 2001. Testing web sites: five users is nowhere near enough. In: Proceedings of CHI '01 extended abstracts on human factors in computing systems, New York: ACM Press, 285-286.
60. Sutcliffe, A. and Gault, B., 2004. Heuristic evaluation of virual reality applications. Interacting with Computers, 16, 831-849.
61. Sutcliffe, A., et al., 2000. Model mismatch analysis: towards a deeper explanation of users' usability problems. Behaviour & Information Technology, 19 (1), 43-55.
62. Tohidi, M., et al., 2006. Getting the right design and the design right. In: Proceedings of ACM conference on human factors in computing. New York: ACM Press, 1243-1252.
63. Tversky, A., 1977. Features of similarity. Psychological Review, 84 (4), 327-352.
64. Weitzenfeld, J., 1984. Valid reasoning by analogy. Philosophy of Science, 51, 137-149.
65. Wixon, D., 2003. Evaluating usability methods: why the current literature fails the practitioner. Interactions, 10 (4), 29-34.
66. Woolrych, A. and Cockton, G., 2001. Why and when five test users aren't enough. In: J.Vanderdonckt,A. Blandford, and A.Derycke, eds. Proceedings of IHM-HCI 2001 conference: Volume 2. Toulouse: CépadèusÉ ditions, 105-108.