USABILITY TESTING: CURRENT PRACTICE AND FUTURE DIRECTIONS

Human-Computer Interaction: Development Process

Volumn , Issue , 2009, Pages 231-252

  Dumas, Joseph S ^a   Fox, Jean E ^b  

^a BENTLEY COLLEGE   (United States)

^b BUREAU OF LABOR STATISTICS   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85138322362     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1201/9781420088892-18     Document Type: Chapter

References (112)

1. Andre, T., Williges, R., and Hartson, H. (2003). The effectiveness of usability evaluation methods: Determining the appropriate criteria. Proceedings of the Human Factors and Ergonomics Society, 43rd Annual Meeting (pp. 1090-1094). Denver, CO.
2. American National Standards Institute. (2001). Common industry format for usability test reports (ANSI-NCITS 354-2001). Washington, DC: Author.
3. Bailey, R. W., Allan, R. W., and Raiello, P. (1992). Usability testing vs. heuristic evaluation: A head-to-head comparison. Proceedings of the Human Factors and Ergonomics Society, 36th Annual Meeting (pp. 409-413). Atlanta, GA.
4. Barker, R. T., and Biers, D. W. (1994). Software usability testing: Do user self-consciousness and the laboratory environment make any difference? Proceedings of the Human Factors Society, 38th Annual Meeting (pp. 1131-1134). Santa Monica, CA.
5. Bojko, A. (2005). Eye tracking in user experience testing: How to make the most of it. Proceedings of the Usability Professionals’ Association (pp. 1-9). Montreal.
6. Bojko, A., and Stephenson, A. (2005). Supplementing conventional usability measures with eye movement data in evaluating visual search performance. Proceedings of the 11th International Conference on Human-Computer Interaction (HCII 2005) (pp. 117-121). Las Vegas, NV.
7. Boren, M., and Ramey, J., (2000). Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication, 43(3), 261-278.
8. Branaghan, R. (1997, April). Ten tips for selecting usability test participants. Common Ground, 7, 3-6.
9. Briggs, J., and Johnson, B. (2005). Practical prototypes: High fidelity website prototypes in use. Paper presented at the Usability Professionals’ Association 14th Annual Conference, Montreal, CA.
10. Brooke, J. (1996). SUS: A quick and dirty usability scale. In P. Jordan, B. Thomas, B. Weerdmeester, and I. McClelland (Eds.), Usability evaluation in industry (pp. 189-194). London: Taylor and Francis.
11. Catani, M. B. (2003). Observation methodologies for usability tests of handheld devices. Proceedings of the Usability Professionals’ Association 12th Annual Conference (pp. 1-6). Scottsdale, AZ.
12. Catani, M. B., and Biers, D. W. (1998). Usability evaluation and prototype fidelity: Users and usability professionals. Proceedings of the Human Factors Society, 42nd Annual Meeting (pp. 1331-1335). Santa Monica, CA.
13. Chin, J. P., Diehl, V. A., and Norman, K. L. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. Proceedings of Human Factors in Computing Systems (CHI’88) (pp. 213-218). Los Angeles, CA.
14. Chisnell, D., Lee, A., and Redish, J. (2005). Recruiting and working with older participants. Washington, DC: AARP. Retrieved October 13, 2005, from http://www.aarp.org/olderwiserwired/oww-features/Articles/a2004-03-03-recruiting-participants.html.
15. Cockton, G., Lavery, D., and Woolrych, A. (2003). Inspection-based evaluations. In J. Jacko and A. Sears (Eds.), The human-computer interaction handbook (pp. 1118-1125). Mahwah, NJ: Lawrence Erlbaum Associates.
16. Coyne, K. P. (2005). Conducting simple usability studies with users with disabilities. Proceedings of HCI International 2005 (pp. 890-893). Las Vegas, NV.
17. Datz-Kauffold, B., and Henry, S. L. (2000). Waving magic wands: Interaction techniques to improve usability testing. Proceedings of the Usability Professionals’ Association 9th Annual Conference (pp. 14-18). Asheville, NC.
18. Desmet, P. (2002). Designing emotion. Unpublished doctoral thesis, Delft University of Technology, Netherlands.
19. Desurvire, H. W. (1994). Faster, cheaper! Are usability inspection methods as effective as empirical testing? In J. Nielsen and R. Mack (Eds.), Usability inspection methods (pp. 173-202). New York: John Wiley and Sons.
20. Desurvire, H. W., Kondziela J. M., and Atwood, M. E. (1992). What is gained and lost when using evaluation methods other than empirical testing. In A. Monk, D. Diaper, and M. D. Harrison (Eds.), People and computers VII (pp. 89-102). Cambridge, UK: Cambridge University Press.
21. Dumas, J. (2001a). Usability testing methods: Think aloud protocols. In R. Branaghan (Ed.), Design by people for people: Essays on usability (pp. 119-130). Chicago, IL: Usability Professionals’ Association.
22. Dumas, J., (2001b). Usability testing methods: How many test participants in a usability test are enough? In R. Branaghan (Ed.), Design by people for people: Essays on usability (pp. 173-182). Chicago, IL: Usability Professionals’ Association.
23. Dumas, J, (2003, November). Usability evaluation from your desktop. Association for Information Systems (AIS) SIGCHI Newsletter, 2(2), 7-8.
24. Dumas, J., and Redish, G. (1993). A practical guide to usability testing. Norwood, NJ: Ablex Publishing Co.
25. Dumas, J., and Redish, G. (1999). A practical guide to usability testing (Rev. ed.). London: Intellect Books.
26. Ebling, M., and John, B. (2000). On the contributions of different empirical data in usability testing. Proceedings of Designing Interactive Systems (pp. 289-296.). Brooklyn, NY.
27. Ericsson, K. A., and Simon, H. A. (1993). Protocol analysis: Verbal reports as Data. Cambridge, MA: MIT Press.
28. Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, and Computers, 35(3), 379-383.
29. Frøkjær, E., Hertzum, M., and Hornbæk, K. (2000). Measuring usability: Are effectiveness, efficiency, and satisfaction really correlated? Proceedings of Human Factors in Computing Systems (CHI 2000) (pp. 345-352). Fort Lauderdale, FL.
30. Gaba, D. M. (1994). Human performance in dynamic medical domains. In M. S. Bogner (Ed.), Human error in medicine (pp. 197-224). Hillsdale, NJ: Lawrence Erlbaum Associates.
31. Goldberg, J. H., Stimson, M. J., Lewenstein, M., Scott, N., and Wichansky, A. M. (2002). Eye tracking in web search tasks: Design implications. Proceedings of the Eye Tracking Research and Applications Symposium (ETRA) (pp. 51-58). New Orleans, LA.
32. Grossnickle, M. M. (2004). How many users with disabilities should you include when conducting a usability test for accessibility? Idea market. Paper presented at The Usability Professionals’ Association Annual Conference, Minneapolis, MN. Retrieved September 13, 2005, from http://www.upassoc.org/usability_resources/conference/2004/im_martinson.html
33. Hancock, P., Pepe, A., and Murphy, L. (2005). Hedonomics: The power of positive and pleasurable ergonomics. Ergonomics in Design, 13(1), 8-14.
34. Hackman, G. S., and Biers, D. W. (1992). Team usability testing: Are two heads better than one? Proceedings of the Human Factors Society, 36th Annual Meeting (pp. 1205-1209). Atlanta, GA.
35. Hanna, L., Risden, K., and Alexander, K. J. (1997, September/October). Guidelines for usability testing with children. Interactions, 9-14.
36. Hassenzahl, M., Beu, A., and Burmester, M. (2001). Engineering joy. IEEE Software, 70-76.
37. Hertzum, M., and Jacobsen, N. E. (2001). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 13(4), 421-443.
38. Hofstede, G. (1997). Cultures and organizations: Software of the mind. New York: McGraw-Hill.
39. Hong, J., and Landay, J. (2001). WebQuilt: A framework for capturing and visualizing the web experience, Proceedings of the 10th International Conference on World Wide Web (pp. 717-724). Seattle, WA.
40. Information Technology Technical Assistance and Training Center, Georgia Institute of Technology. (2004). Planning usability testing for accessibility. Retrieved September 13, 2005, from http://www.ittatc.org/technical/access-ucd/ut_plan.php
41. Jacobsen, N., Hertzum, M., and John, B. E. (1998). The evaluator effect in usability studies: Problem detection and severity judgments. Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting (pp. 1336-1340). Chicago, IL.
42. Jordan, P. (2002). The personalities of products. In W. Green and P. Jordan (Eds.), Pleasure with products (pp. 19-48). London: Taylor and Francis.
43. Kaikkaner, A., Kekalainen, A, Canker, M. Kalliot, T, and Kankainen, A. (2005). Usability testing of mobile applications: A comparison between laboratory and field studies. Journal of Usability Studies, 1, 4-16.
44. Kennedy, S. (1989). Using video in the BNR usability lab. SIGCHI Bulletin, 21, 92-95.
45. Kirakowski, J. (1996). The software usability measurement inventory (SUMI): Background and usage. In P. Jordan, B. Thomas, B. Weerdmeester, and I. McClelland (Eds.), Usability evaluation in industry (pp. 169-177). London: Taylor and Francis.
46. Krahmer, E., and Ummelen, N. (2004). Thinking about thinking aloud: A comparison of two verbal protocols for usability testing, IEEE Transactions on Professional Communication, 47(2), 105-117.
47. Law, C., and Vanderheiden, G. (2000). Reducing sample sizes when user testing with people who have and who are simulating disabilities: Experiences with blindness and public information kiosks. Proceedings of the Human Factors Society, 44th Annual Meeting (pp. 157-160). San Diego, CA.
48. Lepistö, A., and Ovaska, S. (2004). Usability evaluation involving participants with cognitive disabilities. Proceedings of NordiCHI (pp. 305-308). Tampere, Finland.
49. Lesaigle, E. M., and Biers, D. W. (2000). Effect of type of information on real-time usability evaluation: Implications for remote usability testing. Proceedings of the IEA 2000/HFES 2000 Congress (pp. 6, 585-588). San Diego, CA.
50. Lewis, J. (1994). Sample size for usability studies: Additional considerations. Human Factors, 36, 368-378.
51. Lewis, J. (2001). Evaluation of procedures of adjusting problem discovery rates estimates from small samples. International Journal of Human Computer Interaction, 7(1), 57-78.
52. Loring, B. (2002). Conducting bilingual user research. Proceedings of the Usability Professionals’ Association Annual Meeting (pp. 1-5). Orlando, FL.
53. Maslow, A.H. (1970). Motivation and personality (2nd ed.). New York: Viking Press.
54. McGee, M. (2003). Usability magnitude estimation. Proceedings of the Human Factors Society, 47th Annual Meeting (pp. 691-695). Denver, CO.
55. McGee, M. (2004). Master usability scaling: Magnitude estimation and master scaling applied to usability measurement. Proceedings of Human Factors in Computing Systems (CHI 2004) (pp. 335-342). New Orleans, LA.
56. Medlock, M., Wixon, D., McGee, M., and Welsh, D. (2005). The rapid iterative test and evaluation method: Better products in less time. In R. Bias and D. Mayhew (Eds.), Cost-justifying usability: An update for the information age (pp. 489-517). New York: Morgan Kaufman Publishers.
57. Medlock, M. C., Wixon, D., Terrano, M., Romero, R., and Fulton, B. (2002). Using the RITE method to improve products: A definition and a case study. Proceedings of the Usability Professionals’ Association (pp. 1-12). Orlando, FL.
58. Meister, D. (1999). The history of human factors and ergonomics. Mahwah, NJ: Lawrence Erlbaum Associates.
59. Mitropoulos-Rundus, D., and Muszak, J. (1997, January). How to design and conduct a consumer in-home usability test. Common Ground, 7, 10-12.
60. Molich, R, Meghan R. Ede, K., and Karyukin, B. (2004). Comparative usability evaluation. Behaviour and Information Technology, 23, 65-74.
61. Molich, R., Bevan, N., Curson, I., Butler, S., Kindlund, E., Miller, D., et al. (1998). Comparative evaluation of usability tests [Electronic version]. Proceedings of Usability Professionals’ Association, Washington, DC.
62. Molich, R. and Dumas, J. (in press). Comparative usability evaluation (CUE-4), Behaviour and Information Technology.
63. Muñoz, R., Miller-Jacobs, H. H., Spool, J. M., and Verplank, B. (1992). In search of the ideal prototype. Proceedings of Human Factors in Computing Systems (CHI’92) (pp. 577-579). Monterey, CA.
64. Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Proceedings of CHI’92 (pp. 373-380). Monterey, CA.
65. Nielsen, J. (1994). Usability laboratories [Special issue]. Behavior and Information Technology, 13.
66. Nielsen, J. (1996). International usability testing. Retrieved September 13, 2005, from http://www.useit.com/papers/international_usetest.html
67. Nielsen, J., and Levy, J. (1994). Measuring usability: Preference vs. performance. Communications of the ACM, 37, 66-76.
68. Pagulayan, R., Steury, K., Fulton, B., and Romero, R. (2003). Designing for fun: User-testing case studies. In M. Blythe, K. Overbeeke, A. Monk and P. Wright (Eds.), Funology: From usability to enjoyment (pp. 137-150). Norwell, MA: Khuwer Academic Publishers.
69. Pagulayan, R., Keeker, K., Wixon, D., Romero, R., and Fuller, T. (2003). User-centered design in games. In J. Jacko and A. Sears (Eds.), The human-computer interaction handbook (pp. 883-906). Mahwah, NJ: Lawrence Erlbaum Associates.
70. Patel, M., and Paulsen, C. A. (2002). Strategies for recruiting children. Proceedings of the Usability Professionals’ Association Annual Meeting. Orlando, FL.
71. Perkins, R. (2001). Remote usability evaluation over the Internet. In R. Branaghan (Ed.), Design by people for people: Essays on usability (pp. 153-162). Chicago, IL: Usability Professionals’ Association.
72. Pruitt, J., and Adlin, T. (2005). The persona lifecycle: Keeping people in mind throughout product design. San Francisco, CA: Morgan Kaufman.
73. Roberts, T., and Moran, T. (1982). Evaluation of text editors. Proceedings of Human Factors in Computer Systems (pp. 136-141). Gaithersburg, VA.
74. Rosenbaum, S., Rohn, J., and Humburg, J. (2000). A toolkit for strategic usability: Results from workshops, panels, and surveys. Proceedings of CHI 2000 (pp. 337-344). The Hague, Netherlands.
75. Rubin, J. (1994). Handbook of usability testing. New York: John Wiley and Sons.
76. Rudd, J., and Isensee, S. (1994). Twenty-two tips for a happier, healthier prototype. Interactions, 35-40.
77. Rudd, J., Stern, K., and Isensee, S. (1996). Low vs. high-fidelity prototyping debate. Interactions, 76-85.
78. Russell, M. (2005). Using eye-tracking data to understand first impressions of a website. Usability News, 7(1). Retrieved June 23, 2005, from http://psychology.wichita.edu/surl/usabilitynews/71/eye_track ing.html
79. Sauro, J., and Kindlund, E. (2005). A method to standardize usability metrics into a single score. Proceedings of Human Factors in Computing Systems (CHI 2005) (pp. 401-409). Denver, CO.
80. Scholtz, J., and Bouchette, D. (1995, March). Usability testing and groupbased software: Lessons from the field. Common Ground, 5, 1-11.
81. Sears, A. (1997). Heuristic walkthroughs: Finding the problems without the noise. International Journal of Human Computer Interaction, 9, 213-234.
82. Shneiderman, B. (1987). Designing the user interface: Strategies for effective human computer interaction (1st ed.). Reading, MA: Addison-Wesley.
83. Shneiderman, B. (1992). Designing the user interface: Strategies for effective human computer interaction (2nd ed.). Reading, MA: Addison-Wesley.
84. Shneiderman, B. (1997). Designing the user interface: Strategies for effective human computer interaction (3rd ed.). Reading, MA: Addison-Wesley.
85. Skelton, T. M. (1992). Testing the usability of usability testing. Technical Communication, 39(3), 343-359.
86. Skinner, B. F. (1956). A case history in scientific method. American Psychologist, 11, 221-233.
87. Sklar, A., and Gilmore, D. (2004). Are you positive? Interactions, 11, 28-33.
88. Smith, P. A. (1996). Towards a practical measure of hypertext usability. Interacting with Computers, 8(4), 365-381.
89. Snyder, C. (2003). Paper prototyping. New York: Morgan Kaufmann Publishers.
90. Snyder, C. (2004). Paper prototyping grows up. Proceedings of the Society for Technical Communication 51st Annual Conference (pp. 334-338). Salt Lake City, UT.
91. Soderston, C. (1985). The usability edit: A new level. Technical Communication, 32(1), 16-18.
92. Swierenga, S. J., and Guy, T. (2003). Session logistics for usability testing of users with disabilities. The Usability Professionals’ Association Annual Meeting (pp. 1-6). Scottsdale, AZ.
93. Tedesco, D., McNulty, M., and Tullis, T. (2005). Usability testing with older adults. The Usability Professionals’ Association Annual Meeting (pp. 1-8). Montreal, CA.
94. Theofanos, M., and Quesenbery, S. (2005). Towards the design of effective formative test reports. Journal of Usability Studies, 1, 27-45.
95. Thomas, B. (1996). Quick and dirty usability tests. In P. Jordan, B. Thomas, B. Weerdmeester, and I. McClelland (Eds.), Usability evaluation in industry (pp. 107-114). London: Taylor and Francis.
96. Tullis, T., Flieschman, S., McNulty, M., Cianchette, C., and Bergel, M. (2002). An empirical comparison of lab and remote usability testing of websites. The Usability Professionals’ Association Annual Meeting (pp. 1-5). Orlando, FL.
97. Tullis, T., and Stetson, J. (2004). A comparison of questionnaires for assessing website usability. Proceedings of Usability Professionals’ Association (pp. 1-12). Minneapolis, MN.
98. Urokohara, H., Tanaka, K., Furuta, K., Honda, M., and Kurosu, M. (2000). NEM: “Novice Expert ratio Method” A usability evaluation method to generate a new performance measure. Proceedings of Human Factors in Computing Systems (CHI 2000) (pp. 185-186). The Hague, Netherlands.
99. Vatrapu, R., and Pérez-Quiñones, M. A. (2004). Culture and international usability testing: The effects of culture in structured interviews (Technical report cs.HC/0405045). Retrieved October 4, 2005, from Computing Research Repository (CoRR) website: perez.cs.vt.edu/publications.php
100. Virzi, R. A. (1989). What can you learn from a low-fidelity prototype? Proceedings of the Human Factors Society, 33rd Annual Meeting (pp. 224-228). Denver, CO.
101. Virzi, R. A. (1990). Streamlining the design process: Running fewer subjects. Proceedings of the Human Factors Society, 34th Annual Meeting (pp. 291-294). Santa Monica, CA.
102. Virzi, R. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34, 457-468.
103. Virzi, R. A., Sokolov, J. L., and Karis, D. (1996). Usability problem identification using both low- and high-fidelity prototypes. Proceedings of Human Factors in Computing Systems (CHI’96) (pp. 236-243). Vancouver, Canada.
104. Virzi, R. A., Sorce, J. F., and Herbert, L. B. (1993), A comparison of three usability evaluation methods: Heuristic, think-aloud, and performance testing, Proceedings of the Human Factors and Ergonomics Society, 37th Annual Meeting (pp. 309-313). Seattle, WA.
105. Vora, P. (1994, November). Using teaching methods for usability evaluations. Common Ground, 4, 5-9.
106. Walker, M., Takayama, L., and Landay, J. A. (2002). High-fidelity or lowfidelity, paper or computer? Choosing attributes when testing web prototypes. Proceedings of the Human Factors Society, 46th Annual Meeting (pp. 661-665). Atlanta, GA.
107. Waters, S., Carswell, M., Stephens, E., and Selwitz, A. (2001). Research ethics meets usability testing. Ergonomics in Design, 14-20.
108. Wichansky, A. (2000). Usability testing in 2000 and beyond. Ergonomics, 43(7), 998-1006.
109. Wiklund, M. E., Thurrott, C., and Dumas, J. S. (1992). Does the fidelity of software prototypes affect the perception of usability? Proceedings of the Human Factors Society, 36th Annual Meeting (pp. 399-403). Santa Monica, CA.
110. Wilson, C. E., and Coyne, K. P. (2001). Tracking usability issues: To bug or not to bug? Interactions, 8, 15-19.
111. Wolf, C. G. (1989). The role of laboratory experiments in HCI: Help, hindrance or ho-hum? Proceedings of CHI’89 (pp. 265-268). Austin, TX.
112. Young, R., and Barnard, P. (1987). The use of scenarios in HCI research: Turbocharging the tortoise of cumulative science. Proceedings of CHI’87 (pp. 291-296). Toronto, Canada.