Criteria for evaluating usability evaluation methods

International Journal of Human-Computer Interaction

Volumn 15, Issue 1, 2003, Pages 145-181

  Hartson, H Rex ^a,b   Andre, Terence S ^c   Williges, Robert C ^a  

^a Virginia Polytechnic Institute and State University   (United States)

^b VIRGINIA POLYTECHNIC INSTITUTE AND STATE UNIVERSITY   (United States)

^c NONE

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER SOFTWARE; HUMAN COMPUTER INTERACTION; NETWORK PROTOCOLS; SYSTEMS ANALYSIS;

USABILITY EVALUATION METHODS (UEM);

USABILITY ENGINEERING;

EID: 0037242197     PISSN: 10447318     EISSN: None     Source Type: Journal    
DOI: 10.1207/S15327590IJHC1501_13     Document Type: Conference Paper

References (71)

1. Andre, T. S., Belz, S. M., McCreary, F. A., & Hartson, H. R. (2000). Testing a framework for reliable classification of usability problems. In Human Factors and Ergonomics Society Annual Meeting (pp. 573-576). Santa Monica, CA: Human Factors and Ergonomics Society.
2. Andre, T. S., Williges, R. C., & Hartson, H. R. (1999). The effectiveness of usability evaluation methods: Determining the appropriate criteria. In Human Factors and Ergonomics Society 43rd Annual Meeting (pp. 1090-1094). Santa Monica, CA: Human Factors and Ergonomics Society.
3. Bastien, J. M. C., & Scapin, D. L. (1995). Evaluating a user interface with ergonomic criteria. International Journal of Human-Computer Interaction, 7, 105-121.
4. Bastien, J. M. C., Scapin, D. L., & Leulier, C. (1996). Looking for usability problems with the ergonomic criteria and with the ISO 9241-10 dialogue principles. In CHI Conference on Human Factors in Computing Systems (pp. 77-78). New York: ACM.
5. Beer, T., Anodenko, T., & Sears, A. (1997). A pair of techniques for effective interface evaluation: Cognitive walkthroughs and think-aloud evaluations. In Proceedings of the Human Factors and Ergonomics Society 41st Annual Meeting (pp. 380-384). Santa Monica, CA: Human Factors and Ergonomics Society.
6. Bias, R. (1991). Walkthroughs: Efficient collaborative testing. IEEE Software, 8(5), 94-95.
7. Bradford, J. S. (1994). Evaluating high-level design: Synergistic use of inspection and usability methods for evaluating early software designs. In J. Nielsen & R. L. Mack (Eds.), Usability inspection methods (pp. 235-253). New York: Wiley.
8. Card, S. K., Moran, T. P., & Newell, A. (1983). The psychology of human-computer interaction. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
9. Carroll, J. M., Singley, M. K., & Rosson, M. B. (1992). Integrating theory development with design evaluation. Behaviour & Information Technology, 11, 247-255.
10. Chin, J. P., Diehl, V. A., & Norman, K. L. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. In CHI Conference on Human Factors in Computing Systems (pp. 213-218). New York: ACM.
11. Cockton, G., & Lavery, D. (1999). A framework for usability problem extraction. In INTERACT '99 (pp. 344-352). London: IOS Press.
12. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
13. Cuomo, D. L., & Bowen, C. D. (1992). Stages of user activity model as a basis for user-centered interface evaluation. In Annual Human Factors Society Conference (pp. 1254-1258). Santa Monica, CA: Human Factors Society.
14. Cuomo, D. L., & Bowen, C. D. (1994). Understanding usability issues addressed by three user-system interface evaluation techniques. Interacting With Computers, 6(1), 86-108.
15. del Galdo, E. M., Williges, R. C., Williges, B. H., & Wixon, D. R. (1986). An evaluation of critical incidents for software documentation design. In Thirtieth Annual Human Factors Society Conference (pp. 19-23). Anaheim, CA: Human Factors Society.
16. del Galdo, E. M., Williges, R. C., Williges, B. H., & Wixon, D. R. (1987). A critical incident evaluation tool for software documentation. In L. S. Mark, J. S. Warm, & R. L. Huston (Eds.), Ergonomics and human factors (pp. 253-258). New York: Springer-Verlag.
17. Desurvire, H. W., Kondziela, J. M., & Atwood, M. E. (1992). What is gained and lost when using evaluation methods other than empirical testing. In A. Monk, D. Diaper, & M. D. Harrison (Eds.), People and computers Volume VII (pp. 89-102). Cambridge, England: Cambridge University Press.
18. Desurvire, H. W., & Thomas, J. C. (1993). Enhancing the performance of interface evaluators. In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting (pp. 1132-1136). Seattle, WA: Human Factors and Ergonomics Society.
19. Doubleday, A., Ryan, M., Springett, M., & Sutcliffe, A. (1997). A comparison of usability techniques for evaluating design. In Designing interactive Systems (DIS '97) Conference (pp. 101-110). New York: ACM.
20. Dutt, A., Johnson, H., & Johnson, P. (1994). Evaluating evaluation methods. In G. Cockton, S. W. Draper, & G. R. S. Weir (Eds.), People and computers, Volume IX (pp. 109-121). Cambridge, England: Cambridge University Press.
21. Egan, J. P. (1975). Signal detection theory and ROC analysis. New York: Academic.
22. Ericsson, A., & Simon, H. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
23. Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 51, 327-358.
24. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-382.
25. Gray, W. D., & Salzman, M. C. (1998). Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13, 203-262.
26. Hartson, H. R., Andre, T. S., Williges, R. C., & van Rens, L. S. (1999). The user action frame-work: A theory-based foundation for inspection and classification of usability problems. In H.-J. Bullinger & J. Ziegler (Eds.), Human-computer interaction: Ergonomics and user interfaces, Volume 1 (pp. 1058-1062). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
27. Hartson, H. R., & Castillo, J. C. (1998). Remote evaluation for post-deployment usability improvement. In Advanced Visual Interfaces '98 (pp. 22-29). L'Aquila, Italy: ACM Press.
28. Hartson, H. R., Castillo, J. C., Kelso, J., Kamler, J., & Neale, W. C. (1996). Remote evaluation: The network as an extension of the usability laboratory. In CHI Conference on Human Factors in Computing Systems (pp. 228-235). New York: ACM.
29. Hix, D., & Hartson, H. R. (1993a). Developing user interfaces: Ensuring usability through product & process. New York: Wiley.
30. Hix, D., & Hartson, H. R. (1993b). Formative evaluation: Ensuring usability in user interfaces. In L. Bass & P. Dewan (Eds.), Trends in software, Volume 1: User interface software (pp. 1-30). New York: Wiley.
31. Jeffries, R., Miller, J. R., Wharton, C., & Uyeda, K. M. (1991). User interface evaluation in the real world: A comparison of four techniques. In CHI Conference on Human Factors in Computing Systems (pp. 119-124). New York: ACM.
32. John, B. E., & Marks, S. J. (1997). Tracking the effectiveness of usability evaluation methods. Behaviour & Information Technology, 16, 188-202.
33. John, B. E., & Mashyna, M. M. (1997). Evaluating a multimedia authoring tool with cognitive walkthrough and think-aloud user studies. Journal of the American Society of Information Science, 48, 1004-1022.
34. Kahn, M. J., & Prail, A. (1994). Formal usability inspections. In J. Nielsen & R. L. Mack (Eds.), Usability inspection methods (pp. 141-171). New York: Wiley.
35. Karat, C.-M., Campbell, R., & Fiegel, T. (1992). Comparison of empirical testing and walkthrough methods in user interface evaluation. In CHI Conference on Human Factors in Computing Systems (pp. 397-404). New York: ACM.
36. Karat, J. (1998). The fine art of comparing apples and oranges. Human-Computer Interaction, 13, 265-269.
37. Keppel, G. (1991). Design and analysis: A researcher's handbook. Englewood Cliffs, NJ: Prentice Hall.
38. Kies, J. K., Williges, R. C., & Rosson, M. B. (1998). Coordinating computer-supported cooperative work: A review of research issues and strategies. Journal of the American Society for Information Science, 49, 776-779.
39. Landauer, T. K. (1995). The trouble with computers: Usefulness, usability, and productivity. Cambridge, MA: MIT Press.
40. Lavery, D., Cockton, G., & Atkinson, M. P. (1997). Comparison of evaluation methods using structured usability problem reports. Behaviour & Information Technology, 16, 246-266.
41. Lewis, C., Polson, P., Wharton, C., & Rieman, J. (1990). Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces. In CHI Conference on Human Factors in Computing Systems (pp. 235-242). New York: ACM.
42. Lewis, J. R. (1994). Sample sizes for usability studies: Additional considerations. Human Factors, 36, 368-378.
43. Lund, A. M. (1998). Damaged merchandise? Comments on shopping at outlet malls. Human-Computer Interaction, 13, 276-281.
44. Manning, C. D., & Schutze, H. (1999). Foundations of statistical natural language processing. Cambridge, MA: MIT Press.
45. Marchetti, R. (1994). Using usability inspections to find usability problems early in the lifecycle. In Pacific Northwest Software Quality Conference (pp. 1-19). Palo Alto, CA: Hewlett Packard.
46. Meister, D., Andre, T. S., & Aretz, A. J. (1997). System analysis. In T. S. Andre & A. W. Schopper (Eds.), Human factors engineering in system design (pp. 21-55). Dayton, OH: Crew System Ergonomics Information Analysis Center.
47. Monk, A. F. (1998). Experiments are for small questions, not large ones like "What usability evaluation method should I use?" Human-Computer Interaction, 13, 296-303.
48. Neale, D. C., Dunlap, R., Isenhour, P., & Carroll, J. M. (2000). Collaborative critical incident development. In Human Factors and Ergonomics Society 43rd Annual Meeting (pp. 598-601). Santa Monica, CA: Human Factors and Ergonomics Society.
49. Newman, W. M. (1998). On simulation, measurement, and piecewise usability evaluation. Human-Computer Interaction, 13, 316-323.
50. Nielsen, J. (1989). Usability engineering at a discount. In G. Salvendy & M. J. Smith (Eds.), Designing and using human-computer interfaces and knowledge-based systems (pp. 394-401). Amsterdam: Elsevier.
51. Nielsen, J. (1990). Evaluating the thinking aloud technique for use by computer scientists. In H. R. Hartson & D. Hix (Eds.), Advances in human-computer interaction (pp. 69-82). Norwood, NJ: Ablex.
52. Nielsen, J. (1992). Finding usability problems through heuristic evaluation. In CHI Conference on Human Factors in Computing Systems (pp. 373-380). New York: ACM.
53. Nielsen, J. (1994). Heuristic evaluation. In J. Nielsen & R. L. Mack (Eds.), Usability inspection methods (pp. 25-62). New York: Wiley.
54. Nielsen, J., & Mack, R. L. (Eds.). (1994). Usability inspection methods. New York: Wiley.
55. Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. In CHI Conference on Human Factors in Computing Systems (pp. 249-256). New York: ACM.
56. Norman, D. A. (1986). Cognitive engineering. In D. A. Norman & S. W. Draper (Eds.), User centered system design (pp. 31-61). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
57. Olson, G. M., & Moran, T. P. (1998). Commentary on "Damaged merchandise?" Human-Computer Interaction, 13, 263-323.
58. Rubin, J. (1994). Handbook of usability testing. New York: Wiley.
59. Salton, G., & McGill, M. J. (1983). Introduction to modern information retrieval. New York: McGraw-Hill.
60. Scriven, M. (1967). The methodology of evaluation. In R. Tyler, R. Gagne, & M. Scriven (Eds.), Perspectives of curriculum evaluation (pp. 39-83). Chicago: Rand McNally.
61. Sears, A. (1997). Heuristic walkthroughs: Finding the problems without the noise. International Journal of Human-Computer Interaction, 9, 213-234.
62. Smith, S. L., & Mosier, J. N. (1986). Guidelines for designing user interface software (Technical Report No. ESD- TR-86-278/MTR 10090). Bedford, MA: MITRE Corporation.
63. Swets, J. A. (1964). Signal detection and recognition by human observers. New York: Wiley.
64. Thompson, J. A., & Williges, R. C. (2000). Web-based collection of critical incidents during remote usability evaluation. In IEA 2000/HFES 2000 Congress (pp. 602-605). Santa Monica, CA: Human Factors and Ergonomics Society.
65. van Rijsbergen, C. J. (1979). Information retrieval (2nd ed.). London: Butterworths.
66. Virzi, R. A. (1990). Streamlining the design process: Running fewer subjects. In Human Factors and Ergonomics Society 34th Annual Meeting (pp. 291-294). Santa Monica, CA: Human Factors and Ergonomics Society.
67. Virzi, R. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34, 457-468.
68. Virzi, R. A., Sorce, J., & Herbert, L. B. (1993). A comparison of three usability evaluation methods: Heuristic, think-aloud, and performance testing. In Proceedings of the Human Factors and Ergonomics Society 36th Annual Meeting (pp. 309-313). Seattle, WA: Human Factors and Ergonomics Society.
69. Wharton, C., Bradford, J., Jeffries, R., & Franzke, M. (1992). Applying cognitive walkthroughs to more complex user interfaces: Experiences, issues, and recommendations. In CHI Conference on Human Factors in Computing Systems (pp. 381-388). New York: ACM.
70. Winer, B. J., Brown, D. R., & Michels, K. M. (1991). Statistical principles in experimental design. New York: McGraw-Hill.
71. Wright, P., & Monk, A. (1991). A cost-effective evaluation method for use by designers. International Journal of Man-Machine Studies, 35, 891-912.