Integration of usability evaluation studies via a novel meta-analytic approach: What are significant attributes for effective evaluation?

International Journal of Human-Computer Interaction

Volumn 25, Issue 4, 2009, Pages 282-306

  Hwang, Wonil ^a   Salvendy, Gavriel ^b,c  

^a Soongsil University   (South Korea)

^b PURDUE UNIVERSITY   (United States)

^c TSINGHUA UNIVERSITY   (China)

Author keywords

[No Author keywords available]

Indexed keywords

COGNITIVE WALKTHROUGH; EXPERIMENTAL CONDITIONS; HEURISTIC EVALUATION; META-ANALYSIS; TIME CONSTRAINTS; USABILITY EVALUATION; USABILITY EVALUATION METHODS; USABILITY PROBLEMS;

COGNITIVE SYSTEMS; USABILITY ENGINEERING;

EXPERIMENTS;

EID: 70449589332     PISSN: 10447318     EISSN: 15327590     Source Type: Journal    
DOI: 10.1080/10447310802629793     Document Type: Article

References (50)

1. Andre, T. S., Hartson, H. R., & Williges, R. C. (2003). Determining the effectiveness of the usability problem inspector: A theory-based model and tool for finding usability problems. Human Factors, 45, 455-482.
2. Catani, M. B., & Biers, D. W. (1998). Usability evaluation and prototype fidelity: Users and usability professionals. Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting (pp. 1331-1335). Santa Monica, CA: Human Factors and Ergonomics Society.
3. Chattratichart, J., & Brodie, J. (2004). Applying user testing data to UEM performance metrics. Extended Abstracts of the 2004 Conference on Human Factors and Computing Systems (pp. 1119-1122). New York: ACM.
4. Cockton, G., & Woolrych, A. (2001). Understanding inspection methods: Lessons from an assessment of heuristic evaluation. In A. Blandford, J. Vanderdonckt, & P. Gray (Eds.), People and computers (Vol. 15, pp. 171-191). London: Springer.
5. Connell, I. W., & Hammond, N. V. (1999). Comparing usability evaluation principle swith heuristics: Problem instances vs. problem types. In M. Angela Sasse & C. Johnson(Eds.), Proceedings of the IFIP INTERACT '99 Conference on Human-Computer Interaction (pp. 621-629). Amsterdam: IOS.
6. Cordes, R. E. (2001). Task-selection bias: A case for user-defined tasks. Human-Computer Interaction, 13, 411-419.
7. Cuomo, D. L., & Bowen, C. D. (1994). Understanding usability issues addressed by three user-system interface evaluation techniques. Interacting with Computers, 6(1), 86-108.
8. Desurvire, H. W., Kondziela, J. M., & Atwood, M. E. (1992). What is gained and lost when using evaluation methods other than empirical testing. In A. Monk, D. Diaper, &M. D. Harrison (Eds.), People and computers (Vol. 7, pp. 89-102). Cambridge, England: Cambridge University Press.
9. Desurvire, H. W., & Thomas, J. C. (1993). Enhancing the performance of interface evaluators. Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting (pp. 1132-1136. Santa Monica, CA: Human Factors and Ergonomics Society.
10. Doubleday, A., Ryan, M., Springett, M., & Sutcliffe, A. (1997). A comparison of usability techniques for evaluating design. Designing Interactive Systems (DIS '97) Conference Proceedings (pp. 101-110). New York: ACM.
11. Fu, L., Salvendy, G., & Turley, L. (2002). Effectiveness of user testing and heuristic evaluation as a function of performance classification. Behaviour & Information Technology, 21, 137-143.
12. Gray, W. D., & Salzman, M. C. (1998). Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13, 203-261.
13. Hackman, G. S., & Biers, D. W. (1992). Team usability testing: are two heads better than one? Proceedings of the Human Factors Society 36th Annual Meeting, pp. 1205-1209.
14. Harrison, S., & Mancey, P. (1998). Optimal amount of time for obtaining accurate usability-test results. Proceedings of the 16th Annual International Conference on Computer Documentation, pp. 173-179.
15. Hartson, H. R., Andre, T. S., & Williges, R. C. (2003). Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 15(1), 145-181.
16. Hertzum, M., & Jacobsen, N. E. (1999). The evaluator effect during first-time use of the cognitive walk through technique. In H.-J. Bullinger & J. Ziegler (Eds.), Human-computer interaction: Ergonomics and user interfaces (Vol. 1, pp. 1063-1067). London: Lawrence Erlbaum.
17. Hertzum, M., & Jacobsen, N. E. (2001). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 13(4), 421-443.
18. Hertzum, M., Jacobsen, N. E., & Molich, R. (2002). Usability inspections by groups of specialists: Perceived agreement in spite of disparate observations. CHI '02 Extended Abstracts on Human Factors in Computing Systems (pp. 662-663). New York: ACM.
19. Hornbæk, K., & Frøkjær, E. (2004). Two psychology-based usability inspection techniques studied in a diary experiment. Proceedings of the Third Nordic Conference on Human-Computer Interaction (pp. 3-12 New York: ACM.
20. Hornbæk, K., & Frøkjær, E. (2005). Comparing usability problems and redesign proposals as input to practical systems development. Proceedings of the ACM CHI'05 Conference (pp. 391-400). New York: ACM.
21. Jacobsen, N. E., Hertzum, M., & John, B. E. (1998). The evaluator effect in usability studies: Problem detection and severity judgments. Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting (pp. 1336-1340). Santa Monica, CA: Human Factors and Ergonomics Society.
22. Jacobsen, N. E., & John, B. E. (2000). Two case studies in using cognitive walkthrough for interface evaluation (CMU Tech. Rep. No. CMU-CS-00-132). Pittsburgh, PA: Carnegie Mellon University.
23. John, B. E., & Mashyna, M. M. (1997). Evaluating a multimedia authoring tool with cognitive walk through and think-aloud user studies. Journal of the American Society of Information Science, 48, 1004-1022.
24. Karat, C.-M., Campbell, R., & Fiegel, T. (1992). Comparison of empirical testing and walk through methods in user interface evaluation. CHI Conference on Human Factors in Computing Systems (pp. 397-404). New York: ACM.
25. Kjeldskov, J., & Skov, M. B. (2003). Creating realistic laboratory settings: Comparative studies of three think-aloud usability evaluations of a mobile system. Proceedings of the 9th IFIPTC13 International Conference on Human-Computer Interaction, INTERACT 2003 (pp. 663-670).Zürich: IOS Press.
26. Kjeldskov, J., Skov, M. B., & Stage, J. (2004). Instant data analysis: Conducting usability evaluations in a day. Proceedings of the Third Nordic Conference on Human-Computer Interaction (pp. 233-240). New York: ACM.
27. Lavery, D., Cockton, G., & Atkinson, M. P. (1997). Comparison of evaluation methods using structured usability problem reports. Behaviour & Information Technology, 16(4/5),246-266.
28. Law, L.-C., & Hvannberg, E. T. (2002). Complementarity and convergence of heuristic evaluation and usability test: A case study of universal brokerage platform. Proceedings of the Second Nordic Conference on Human-Computer Interaction (pp. 71-80). New York: ACM.
29. Law, L.-C., & Hvannberg, E. T. (2004). Analysis of combinatorial user effect in international usability tests. CHI Conference on Human Factors in Computing Systems (pp. 9-16).New York: ACM.
30. Lewis, C. (1982). Using the "thinking-aloud" method in cognitive interface design (Res. Rep. No.RC9265). Yorktown Heights, NY: IBM T. J. Watson Research Center.
31. Lewis, C., Polson, P., Wharton, C., & Rieman, J. (1990). Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces. Proceedings of the ACM CHI '90Conference (pp. 235-242). New York: ACM.
32. Lewis, J. R. (1994). Sample sizes for usability studies: Additional considerations. Human Factors, 36(2), 368-378.
33. Lewis, J. R. (2001). Evaluation procedures for adjusting problem-discovery rates estimated from small samples. International Journal of Human-Computer Interaction, 13(4), 445-479.
34. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
35. Mankoff, J., Dey, A. K., Hsieh, G., Kientz, J., Lederer, S., & Ames, M. (2003). Heuristic evaluationof ambient displays. CHI Conference on Human Factors in Computing Systems (pp. 169-176). New York: ACM.
36. Nielsen, J. (1990). Evaluating the thinking aloud technique for use by computer scientists. In H. R. Hartson & D. Hix (Eds.), Advances in human-computer interaction (pp. 69-82).Norwood, NJ: Ablex.
37. Nielsen, J. (1992). Finding usability problems through heuristic evaluation. CHI Conference on Human Factors in Computing Systems (pp. 373-380). New York: ACM.
38. Nielsen, J. (1994). Estimating the number of subjects needed for a thinking aloud test. International Journal of Human-Computer Studies, 41, 385-397.
39. Nielsen, J. (1997). Usability testing. In G. Salvendy (Ed.), Handbook of human factors and ergonomics(2nd ed., pp. 1543-1568). New York: John Wiley & Sons.
40. Nielsen, J., & Landauer, T. K. (1993). A mathematical model of the finding of usability problems. Proceedings of the ACM INTERCHI '93 Conference (pp. 206-213). New York: ACM.
41. Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interface. In CHI '90 Conference Proceedings (pp. 249-256), New York: ACM.
42. Polson, P., Lewis, C., Rieman, J., & Wharton, C. (1992). Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. International Journal of Man-Machine Studies,36, 741-773.
43. Sears, A. (1997). Heuristic walkthroughs: Finding the problems without the noise. International Journal of Human-Computer Interaction, 9, 213-234.
44. Sears, A., & Hess, D. J. (1999). Cognitive walkthroughs: Understanding the effect of task description detail on evaluator performance. International Journal of Human-Computer Interaction, 11, 185-200.
45. Virzi, R. A. (1990). Streamlining the design process: Running fewer subjects. Human Factors Society 34th Annual Meeting (pp. 291-294). Santa Monica, CA: Human Factors and Ergonomics Society.
46. Virzi, R. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34, 457-468.
47. Virzi, R. A., Sokolov, J. L., & Karis, D. (1996). Usability problem identification using both low- and high-fidelity prototypes. CHI Conference on Human Factors in Computing Systems (pp. 236-243). New York: ACM.
48. Virzi, R. A., Sorce, J., & Herbert, L. B. (1993). A comparison of three usability evaluation methods: Heuristic, think-aloud, and performance testing. Proceedings of the Human Factors Society 36th Annual Meeting (pp. 309-313). Santa Monica, CA: Human Factors and Ergonomics Society.
49. Wixon, D. (2003). Evaluating usability methods: Why the current literature fails the practitioner. interactions, 10(4), 28-34.
50. Wright, P. C., & Monk, A. F. (1991). A cost-effective evaluation method for use by designers. International Journal of Man-Machine Studies, 35, 891-912.