Comparison of evaluation methods using structured usability problem reports

Behaviour and Information Technology

Volumn 16, Issue 4-5, 1997, Pages 246-266

  Lavery, Darryn ^a   Cockton, Gilbert ^b   Atkinson, Malcolm P ^a  

^a UNIVERSITY OF GLASGOW   (United Kingdom)

^b NORTHUMBRIA UNIVERSITY   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER SCIENCE; DESIGN; EVALUATION; HUMAN COMPUTER INTERACTION;

USABILITY PROBLEM;

INFORMATION TECHNOLOGY;

EID: 0031188474     PISSN: 0144929X     EISSN: 13623001     Source Type: Journal    
DOI: 10.1080/014492997119824     Document Type: Article

References (51)

1. Atkinson, M. P. and Morrison, R. 1995, Orthogonal persistent object systems, Journal of Very Large Databases, 4,319-401.
2. Bastien, J. M. C. and Scapin, D. L. 1995, Evaluating a user interface with ergonomic criteria, International Journal of Human-Computer Interaction, 7, 105-121.
3. Bell, B., Rieman, J. and Lewis, C. 1991, Usability testing of a graphical programming system: things we missed in a programming walkthrough, in S. P. Robertson, G. M. Olson and J. S. Olson (eds), Proceedings of ACM CHI’91 Conference on Human Factors in Computing Systems (ACM, New York), 7-12.
4. Card, S. K. and Moran, T. P. 1980, The Keystroke-Level Model for User Performance Time with Interactive Systems. Communications of the ACM, 23, 396-410.
5. Card, S. K., Moran, T. P. and Newell, A. 1983. The Psychology of Human-Computer Interaction (Lawrence Erlbaum Associates, Hillsdale, NJ).
6. Carroll, J. M., Koenemann-Belliveau, J., Rosson, M. B. and Singley, M. K. 1993, Critical incidents and critical themes in empirical usability evaluation, in J. L. Alty, D. Diaper and 5. Guest (eds), Proceedings of the HCI‘93 Conference on People and Computers VIII (Cambridge University Press, Cambridge), 279-292.
7. Cuomo, D. L. and Bowen, C. D. 1992, Stages of user activity model as a basis for user-system interface evaluations, in Proceedings of the Human Factors Society 36th Annual Meeting (Human Factors Society, Santa Monica), 1254-1258.
8. Cuomo, D. L. and Bowen, C. D. 1994, Understanding usability issues addressed by three user-system interface evaluation techniques, Interacting with Computers, 6, 86-108.
9. Desurvire, H. W. 1994, Faster, Cheaper!! Are usability inspection methods as effective as empirical testing? in J. Nielsen and R. L. Mack (eds), Usability Inspection Methods (John Wiley and Sons, New York), 173-202.
10. D esurvire, H. W., Kondziela, J. M. and Atwood, M. E. 1992, what is gained and lost when using evaluation methods other than empirical testing, in A. Monk, D. Diaper and M. Harrison (eds), Proceedings of the HCI‘92 Conference on People and Computers VII (Cambridge University Press, Cambridge), 89-102.
11. Dowell, J., Salter, I. and Zekrullahi, S. 1994, A domain analysis of air traffic management work can be used to rationalise interface design issues, in G. Cockton, S. W. Draper and G. R. S. Weir (eds), People and Computers IX. Proceedings of BCS HCI‘94 (Cambridge University Press, Cambridge), 53-65.
12. Dutt, A., Johnson, H. and Johnson, P. 1994, Evaluating evaluation methods, in G. Cockton, S. W. Draper and G. R. S. Weir (eds), People and Computers IX. Proceedings of BCS HCI‘94 (Cambridge University Press, Cambridge), 109-121.
13. Fischer, G. 1993, Beyond human computer interaction: designing useful and usable computational environments, in J. L. Alty, D. Diaper and S. Guest (eds), Proceedings of the HCI‘93 Conference on People and Computers VIII (Cambridge University Press, Cambridge), 17-31.
14. Green T. R. G. 1990, The cognitive dimension of viscosity: a sticky problem for HCI, in D. Diaper, D. Gilmore, G. Cockton and B. Shackel (eds), Proceedings of IFIP INTERACT'90: Human-Computer Interaction (Elsevier Science, North-Holland), 79-86.
15. Green, T. R. G. and Petre, M. 1996, Usability analysis of visual programming environments: a ‘cognitive dimensions’ framework, Journal of Visual Languages and Computing, 7, 131-174.
16. Hollnagel, E. 1993, The phenotype of Erroneous Actions, International Journal of Man-Machine Studies, 39, 1-32.
17. Jeffries, R. 1994, Usability problem reports: helping evaluators communicate effectively with developers, in J. Nielsen and R. L. Mack (eds), Usability Inspection Methods (John Wiley and Sons, New York), 273-294.
18. Jeffries, R., Miller, J. R., Wharton, C. and Uyeda, K. M. 1991, User interface evaluation in the real world: a comparison of four techniques, in S. P. Robertson, G. M. Olson and J. S. Olson (eds), Proceedings of ACM CHI‘91 Conference on Human Factors in Computing Systems (ACM, New York), 119-124.
19. John, B. E. 1990, Extensions of GOMS analyses to expert performance requiring perception of dynamic visual and auditory information, in J. C. Chew and J. Whiteside (eds), Proceedings of A CM CHI‘90 Conference on Human Factors in Computing Systems (ACM, New York), 107-115.
20. John, B. E. and Mashyna, M. M. (1997), Evaluating a multimedia authoring tool with cognitive walkthrough and think-aloud user studies, Journal of the American Society of Information Science. 48 (9).
21. John, B. E. and Packer, H. 1995, Learning and using the cognitive walkthrough method: a case study approach, in I. Katz, R. Mack and L. Marks (eds), Proceedings of ACM CHI‘95 Conference on Human Factors in Computing Systems (ACM, New York), 429-436.
22. John, B. E. and Vera, A. H. 1992, A GOMS analysis of a graphic, machine-paced, highly interactive task, in P. Bauersfeld, J. Bennett and G. Lynch (eds), Proceedings of A CM CHI‘92 Conference on Human Factors in Computing Systems (ACM, New York), 251-258.
23. Kahn, M. J. and Prail, A. 1994, Formal usability inspections, in J. Nielsen and R. L. Mack (eds), Usability Inspection Methods (John Wiley and Sons, New York), 141-171.
24. Karat, C., Campbell, R. and Fiegel, T. 1992, Comparison of empirical testing and walkthrough methods in user interface evaluation, in P. Bauersfeld, J. Bennett and G. Lynch (eds), Proceedings of ACM CHI‘92 Conference on Human Factors in Computing Systems (ACM, New York), 397-404.
25. Lavery, D. 1995a, The Design of Effective Software Visualizations for Persistent Programming Languages, FIDE Technical Report FIDE/95/116, University of Glasgow.
26. Lavery, D. 1995b, A Napier88 Programmer Study, FIDE Technical Report FIDE/95/140, University of Glasgow.
27. Lavery, D. and Cockton, G. 1995, A Pilot Study of Early Usability Evaluation Methods for Software Visualisations, FIDE Technical Report FIDE/95/141, University of Glasgow.
28. Lavery, D. and Cockton, G. 1996, Iterative development of early usability evaluation methods for software visualisations, in W. D. Gray and D. A. Boehm-Davis (eds), Proceedings of the 6th Workshop of Empirical Studies of Programmers (Ablex, Norwood, New Jersey), 275-276.
29. Lewis, C., Polson, P., Wharton, C. and Rieman, J. 1990, Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces, in J. C. Chew and J. Whiteside (eds), Proceedings of ACM CHP90 Conference on Human Factors in Computing Systems (ACM, New York), 235-242.
30. Mack, R. and Montaniz, F. 1994, Observing, predicting and analyzing usability problems, in J. Nielsen and R. L. Mack (eds), Usability Inspection Methods (John Wiley and Sons, New York), 295-339.
31. Mackinlay, J. 1986, Automating the design of graphical presentations of relational information, ACM Transactions on Graphics, 5, 110-141.
32. Maissel, J., Macleod, M., Thomas, C., Renger, R., Corcoran, R., Dillon, A., Maguire, M. and Sweeney, M. 1993, Usability Context Analysis: A practical Guide, Version 3.0, (National Physical Laboratory, Teddington, UK).
33. Modugno, F., Green, T. R. G. and Myers, B. A. 1994, Visual programming in a visual domain: A case study of cognitive dimensions, in G. Cockton, S. W. Draper and G. R. S. Weir (eds), People and Computers IX. Proceedings of BCS HCP94 (Cambridge University Press, Cambridge), 91-108.
34. Morrison, R., Brown, A. L., Connor, R. C. H., Cutts, Q. I., Dearle, A., Kirby, G. N. C. and Munro, D. S. 1994, The Napier88 Reference Manual Release 2.0, Technical Report FIDE/94/104, St. Andrews University.
35. M uller, M. J., Dayton, T. and Root, R. 1993, Comparing studies that compare usability assessment methods: an unsuccessful search for stable criteria, in S. Ashlund, K. Mullet, A. Henderson, E. Hollnagel and T. White (eds), Proceedings of ACM INTERCHI‘93 Conference on Human Factors in Computing Systems-Adjunct Proceedings (ACM, New York), 185-186.
36. Myers, B. A. 1990, Taxonomies of visual programming and program visualization, Journal of Visual Languages and Computing, 1, 97-123.
37. Nielsen, J. 1992, Finding usability problems through heuristic evaluation, in P. Bauersfeld, J. Bennett and G. Lynch (eds), Proceedings of A CM CHI‘92 Conference on Human Factors in Computing Systems (ACM, New York), 373-380.
38. Nielsen, J. 1994, Heuristic evaluation, in J. Nielsen and R. L. Mack (eds), Usability Inspection Methods (John Wiley and Sons, New York), 25-62.
39. Nielsen, J. and Mack, R. L. 1994, Usability Inspection Methods (John Wiley & Sons, New York).
40. Nielsen, J. and Molich, R. 1990, Heuristic evaluation of user interfaces, in J. C. Chew and J. Whiteside (eds), Proceedings of A CM CHI‘90 Conference on Human Factors in Computing Systems (ACM, New York), 249-256.
41. Norman, D. A. 1986, Cognitive engineering, in D. A. Norman and S. W. Draper (eds), User Centered System Design: New Perspectives on Human-Computer Interaction (Lawrence Erlbaum Associates, Hillsdale, NJ), 31-61.
42. Norman, D. A. 1991, Cognitive artifacts, in J. M. Carroll (ed), Designing Interaction: Psychology at the Human-Computer Interface (Cambridge University Press, Cambridge), 17-38.
43. Payne, S. J. and Green, T. R. G. 1986, Task-Action Grammars: A model of the Mental Representation of Task Languages, Human-Computer Interaction, 2, 93-133.
44. Polson, P. G., Lewis, C., Rieman, J. and Wharton, C. 1992, Cognitive walkthroughs: a method for theory-based evaluation of user interfaces, International Journal of Man-Machine Studies, 36, 741-773.
45. Polson, P. G. and Lewis, C. H. 1990, Theory-based design for easily learned interfaces, Human-Computer Interaction, 5, 191-220.
46. Smith, S. L. and Mosier, J. N. 1986, Guidelines for Designing User Interface Software, Technical Report ESD-TR-86-278, The MITRE Corporation, Bedford, MA 01730.
47. Stasko, J., Badre, A. and Lewis, C. 1993, Do algorithm animations assist learning? an empirical study and analysis, in S. Ashlund, K. Mullet, A. Henderson, E. Hollnagel and T. White (eds), Proceedings of ACM INTERCHI‘93 Conference on Human Factors in Computing Systems (ACM, New York), 61-66.
48. Tufte, E. R. 1983, The Visual Display of Quantitative Information (Graphics Press, Cheshire, CT).
49. van Schaik, P., Edwards, J. and Petrie, H. 1995, SATURN Project: Deliverable 3c: An initial evaluation of an existing ATM, Psychology Division, University of Hertfordshire.
50. Wharton, C., Rieman, J., Lewis, C. and Polson, P. 1994, The cognitive walkthrough: a practitioner’s guide, in J. Nielsen and R. L. Mack (eds), Usability Inspection Methods (John Wiley and Sons, New York), 105-140.
51. Wilde, N. P. 1996, Using cognitive dimensions in the classroom as a discussion tool for visual language design, in M. Tauber (ed), Proceedings of ACM CHI‘96 Conference on Human Factors in Computing Systems-Conference Companion (ACM, New York), 187-188.