Evaluating the downstream utility of user tests and examining the developer effect: A case study

International Journal of Human-Computer Interaction

Volumn 21, Issue 2, 2006, Pages 147-172

  Law, Effie Lai Chong ^a  

^a ETH ZURICH   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

DEVELOPMENT TEAMS; INFORMATION INTEGRATION; MULTI-PERSPECTIVE; RESEARCH QUESTIONS; USABILITY EVALUATION; USABILITY PROBLEMS; USER TESTS;

DIGITAL LIBRARIES; USABILITY ENGINEERING;

RESEARCH;

EID: 33751514337     PISSN: 10447318     EISSN: None     Source Type: Journal    
DOI: 10.1207/s15327590ijhc2102_3     Document Type: Review

References (44)

1. Anderson, N. H. (1981). Foundations of information integration theory. New York: Academic.
2. Anderson, N. H. (1982). Methods of information integration theory. New York: Academic.
3. Anderson, N. H. (Ed.). (1991). Contributions to information integration theory (Vols. 1, 2, and 3). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
4. Anderson, N. H. (1996). A functional theory of cognition. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
5. Andre, T. S., Hartson, H. R., Beiz, S. M., & McCreary, F. A. (2001). The user action framework: A reliable foundation for usability engineering support tools. International Journal of Human-Computer Studies, 54, 107-136.
6. Artim, J. M. (2003). Usability problem severity ratings. Retrieved November 17, 2004, from http://www.primaryview.org/CommonDefinitions/
7. Bailey, B. (2005). Judging the severity of usability issues on Web sites: This doesn't work. Retrieved November 11, 2005, from http://www.usability.gov/ pubs/102005news.html
8. Bailey, R. W., Allan, R. W., & Raiello, P. (1992). Usability testing vs. heuristic evaluation: A head-to-head comparison. Proceedings of Human Factors Society 36th Annual Meeting, 409-413.
9. Bevan, N., & Kincla, S. (2004). HCI Design Foundation Study: Final report. Retrieved November 20,2005, from http://www.jisc.ac.uk/ uploaded_documents/JISC-HCIDesign-Study-Final.doc
10. Bolchini, D., & Colazzo, S. (2005). Guidelines for describing usability problems. Proceedings of HCI International 2005 (CD ROM). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
11. Book reviews. (1994). American Journal of Psychology, 107, 441-454.
12. Chattratichart, J., & Brodie, J. (2004). Applying user testing data to UEM performance metrics. Proceedings of CHI 2004, 1119-1122.
13. Cockton, G. (2005). I can't get no iteration. Interfaces, 63, 4.
14. Constantine, L. (2003). Testing ... 1 2 3 4 5 ... Testing .... Retrieved January 2, 2005, from http://www.usabilitynews.com/news/article1058.asp
15. Doubleday, A., Ryan, M., Springett, M., & Sutcliffe, A. (1997). A comparison of usability techniques for evaluating design. Proceedings of DIS' 97, 101-110.
16. Dumas, J. S. (2003). User-based evaluation. In J. A. Jacko & A. Sears (Eds.), Handbook of human-computer interaction (pp. 1093-1117). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
17. Dumas, J. S., & Redish, J. C. (1999). A practical guide to usability testing (Rev. ed.). Exeter, UK: Intellect.
18. Eagly, A., & Chaiken, S. (1993). Psychology of attitudes. New York: Harcourt, Brace Jovanovich.
19. Festinger, L. A. (1957). A theory of cognitive dissonance. Stanford, CA: Stanford University Press.
20. Frøkjær, E., & Hornbæk, K. (2005). Cooperative usability testing: Complementing usability tests with user-supported interpretation sessions. CHI 2005 Extended Abstracts, 1383-1386.
21. Gray, W. D., & Salzman, M. C. (1998). Damaged merchandise? Human-Computer Interaction, 13, 203-262.
22. Hartson, H. R., Andre, T. S., & Williges, R. C. (2001). Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 13, 373-410.
23. Hassenzahl, M., Platz, A., Burmester, M., & Lehner, K. (2000). Hedonic and ergonomic quality aspects determine a software's appeal. Proceedings of CHI 2000, 201-208.
24. Hertzum, M., & Jacobsen, N. E. (2001). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 13, 421-443.
25. Hornbæk, K., & Frøkjær, E. (2005). Comparing usability problems and redesign proposals as input to practical systems development. Proceedings of CHI 2005, 391-400.
26. Jeffries, R., Miller, J. R., Wharton, C., & Uyeda, K. M. (1991). User interface evaluation in the real world: A comparison of four techniques. Proceedings of CHI'91, 119-124.
27. John, B. (1998). On our case study of claims analysis and other usability evaluation methods. Behaviour and Information Technology, 17, 244-246.
28. John, B., & Marks, S. J. (1997). Tracking the effectiveness of usability evaluation method. Behaviour and Information Technology, 16(4/5), 188-202.
29. John, B., & Mashyna, M. M. (1997). Evaluating a multimedia authoring tool. Journal of the American Society for Information Science, 48, 1004-1022.
30. Karat, C.-M., Campbell, R. L., & Fiegel, T. (1992). Comparison of empirical testing and walkthrough methods in user interface evaluation. Proceedings of CHI 92, 397-404.
31. Law, L.-C., & Hvannberg, E. T. (2002). Complementarity and convergence of heuristic evaluation and usability test: A case study of UNIVERSAL brokerage platform. Proceedings of NordiCHI 2002, 71-80.
32. Law, E. L.-C., & Hvannberg, E. T. (2004). Analysis of combinatorial user effect in international usability tests. Proceedings of CHI 2004, 9-16.
33. Lewis, J. R. (1994). Sample sizes for usability studies: Additional considerations. Human Factors, 36, 368-378.
34. Medlock, C. M., Wixon, D., Terrano, M., Romero, R. L., & Fulton, B. (2002). Using the RITE method to improve products; a definition and a case study. Proceedings of UPA'02. Retrieved January 2,2005 from http://www. microsoft.com/usability/publications.html.
35. Molich, R., Ede, M. R., Kaasgaard, K., & Karyukin, B. (2004). Comparative usability evaluation. Behavior and Information Technology, 23, 65-74.
36. Pagulayan, R. J., Keeker, K., Wixon, D., Romero, R., & Fuller, T. (2003). User-centered design in games. In J. Jacko & A. Sears (Eds.), Handbook for human-computer interaction in interactive systems (pp. 883-906). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
37. Sawyer, P., Flanders, A., & Wixon, D. (1996). Making a difference-The impact of inspections. Proceedings of CHI'96, 376-382.
38. Sears, A. (1997). Heuristic walkthroughs. Journal of Human-Computer Interaction, 9, 213-234.
39. Simon, H. A. (1976). Discussion: Cognition and social behavior. In J. S. Carroll & J. W. Payne (Eds.), Cognition and social behavior (chap. 16). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
40. Wilkening, F. (1979). Combining of stimulus dimensions in children's and adults' judgments of area: An information integration analysis. Developmental Psychology, 15,25-33.
41. Wixon, D. R. (1995). Book review on "Cost justifying usability." SIGCHI Bulletin, 27, 94-98.
42. Wixon, D. R (2003). Evaluating usability methods: Why the current literature fails the practitioner. Interactions, 10(4), 29-34.
43. Wixon, D. R., & Jones, S. (1995). Usability for fun and profit: A case study of the design of DEC Rally Version 2. In M. Rudisill, C. Lewis, P. B. Polson, & T. D. McKay (Eds.), Human computer interaction: Success stories, emerging methods and real world context (pp. 3-35). San Francisco: Morgan Kaufmann.
44. Yin, R. (1994). Case study research: Design and methods (2nd ed.). Beverly Hills, CA: Sage.