Reviewing and extending the five-user assumption: A Grounded Procedure for interaction evaluation

ACM Transactions on Computer-Human Interaction

Volumn 20, Issue 5, 2013, Pages

  Borsci, Simone ^a   Macredie, Robert D ^a   Barnett, Julie ^a   Martin, Jennifer ^b   Kuljis, Jasna ^a   Young, Terry ^a  

^a BRUNEL UNIVERSITY   (United Kingdom)

^b UNIVERSITY OF NOTTINGHAM   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

IN-FIELD; INFORMATION APPLICATIONS; INTERACTION EVALUATIONS; INTERACTION PROBLEMS; INTERACTION TESTING; MEDICAL DEVICE DESIGN;

HUMAN COMPUTER INTERACTION;

BIOMEDICAL EQUIPMENT;

EID: 84887850239     PISSN: 10730516     EISSN: 15577325     Source Type: Journal    
DOI: 10.1145/2506210     Document Type: Article

References (47)

1. Alshamari, M. and Mayhew, P. 2009. Tech. Review: Current Issues of Usability Testing. IETE Tech. Rev. 26, 402-406.
2. Bias, R. G. and Mayhew, D. J. 2005. Cost-Justifying Usability: An Update for the Internet Age. Morgan Kaufmann Publishers, San Francisco, CA.
3. Borsci, S., Federici, S., Mele, M. L., Polimeno, D., and Londei, A. 2012. The bootstrap discovery behaviour model: Why five users are not enough to test user experience. In Cognitively Informed Intelligent Interfaces: Systems Design and Development, E. M. Alkhalifa and K. Gaid Eds., IGI Global Press, Hershey, PA.
4. Borsci, S., Londei, A., and Federici, S. 2011. The Bootstrap Discovery Behaviour (BDB): A new outlook on usability evaluation. Cognit. Process. 12, 23-31.
5. Caulton, D. A. 2001. Relaxing the homogeneity assumption in usability testing. Behav. Infor. Technol. 20, 1-7.
6. Crystal, A. and Greenberg, J. 2005. Usability of a metadata creation application for resource authors. Library Inf. Sci. Res. 27, 177-189.
7. Efron, B. 1979. Bootstrap methods: Another look at the jackknife. Ann. Stat. 7, 1-26.
8. Faulkner, L. 2003. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behav. Res. Meth. 35, 379-383.
9. Federici, S., Borsci, S., and Stamerra, G. 2010. Web usability evaluation with screen reader users: Implementation of the Partial Concurrent Thinking Aloud technique. Cognit. Process. 11, 263-272.
10. Fishman, G. S. 1995. Monte Carlo: Concepts, Algorithms, and Applications. Springer, New York.
11. Food and Drug Administration (FDA) 2011. Draft guidance for industry and Food and Drug Administration staff - Applying human factors and usability engineering to optimize medical device design. U. S. Food and Drug Administration, Silver Spring, MD.
12. Fox, J. 2002. An R and S-Plus Companion to Applied Regression. SAGE.
13. Glaser, B. G. and Strauss, A. L. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine Publishing Company, Chicago, IL.
14. Good, I. J. 1953. The population frequencies of species and the estimation of population parameters. Biometrika 40, 237-264.
15. Hertzum, M. and Jacobsen, N. E. 2003. The evaluator effect: A chilling fact about usability evaluation methods. Int. Jo. Hum.-Comput. Interact. 15, 183-204.
16. Hong, J. I., Heer, J., Waterson, S., and Landay, J. A. 2001. WebQuilt: A proxy-based approach to remote web usability testing. ACM Trans. on Inf. Syst. 19, 263-285.
17. ISO 1998. ISO 9241-11:1998, Ergonomic requirements for office work with visual display terminals, CEN, Brussels, BE.
18. Jelinek, F. 1997. Statistical Methods for Speech Recognition. MIT Press, Cambridge, MA.
19. Kirakowski, J. 2005. Chapter 18 - Summative usability testing: Measurement and sample size. In Cost-Justifying Usability 2nd Ed., R. G. Bias and D. J. Mayhew Eds., Morgan Kaufmann, San Francisco, CA, 519-553.
20. Kurosu, M. 2007. Concept of usability revisited. In Human-Computer Interaction: Interaction Design and Usability, J. Jacko Ed., Springer, Berlin, 579-586.
21. Lewis, J. R. 1994. Sample sizes for usability studies: Additional considerations. Human Factors: J. Human Factors Ergono. Soc. 36, 368-378.
22. Lewis, J. R. 2000. Validation of Monte Carlo estimation of problem discovery likelihood. Tech. rep. 29.3357, IBM, Raleigh, NC.
23. Lewis, J. R. 2001. Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. Int. J. Hum.-Comput. Interact. 13, 445-479.
24. Lewis, J. R. 2006. Sample sizes for usability tests: Mostly math, not magic. Interactions 13, 29-33.
25. Lindgaard, G. and Chattratichart, J. 2007. Usability testing: What have we overlooked? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, 1415-1424.
26. Manning, C. D. and Schutze, H. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA.
27. Molich, R. and Nielsen, J. 1990. Improving a human-computer dialogue. Comm. ACM 33, 338-348.
28. Nielsen, J. 1995. Severity ratings for usability problems. http://useit.com/papers/heuristic/severityrating.html.
29. Nielsen, J. 2000. Why you only need to test with 5 users. http://www.useit.com/alertbox/20000319.html.
30. Nielsen, J. 2012. How many test users in a usability study? http://www.useit.com/alertbox/number-of-test-users.html.
31. Nielsen, J. and Landauer, T. K. 1993. A mathematical model of the finding of usability problems. In Proceedings of the INTERACT'93 and CHI'93 Conference on Human Factors in Computing Systems, ACM, New York, 206-213.
32. Nielsen, J. and Mack, R. L. 1994. Usability Inspection Methods. Wiley, New York.
33. Norman, D. A. 1983. Some observations on mental models. In Mental Models, D. Gentner and A. Steven Eds., Lawrence Earlbaum Associates, Hillsdale, NJ, 7-14.
34. Norman, D. A. 1988. The Psychology of Everyday Things. Basic Books, New York.
35. Norman, D. A. and Draper, S. W. 1986. User Centered System Design: New Perspectives on Human-Computer Interaction. Lawrence Erlbaum Associates Inc, Hillsdale, NJ.
36. Petrie, H. and Bevan, N. 2009. The evaluation of accessibility, usability, and user experience. In The Universal Access Handbook, C. Stephanidis Ed., CRC Press, London, UK.
37. Sauro, J. and Lewis, J. R. 2012. Quantifying the User Experience. Morgan Kaufmann, Waltham, MA.
38. Schmettow, M. 2008. Heterogeneity in the usability evaluation process. In Proceedings of the 22nd British HCI Group Annual Conference on People and Computers: Culture, Creativity, Interaction. Vol. 1, UK British Computer Society, Swinton, UK, 89-98.
39. Schmettow, M. 2009. Controlling the usability evaluation process under varying defect visibility. In Proceedings of the 23rd British HCI Group Annual Conference on People and Computers: Celebrating People and Technology. British Computer Society, Swinton, UK, 188-197.
40. Schmettow, M. 2012. Sample size in usability studies. Comm. ACM 55, 64-70.
41. Shneiderman, B. 1986. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Addison-Wesley Longman Publishing Co., Inc, Boston, MA.
42. Spool, J. and Schroeder, W. 2001. Testing web sites: Five users is nowhere near enough. In Proceedings of the CHI'01 Extended Abstracts on Human Factors in Computing Systems. New York, 285-286.
43. Tullis, T. and Albert, W. 2008. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Morgan, Kaufmann.
44. Turner, C. W., Lewis, J. R., and Nielsen, J. 2006. Determining usability test sample size. In International Encyclopedia of Ergonomics and Human Factors, W. Karwowski Ed., CRC Press, Boca Raton, FL, 3084-3088.
45. Virzi, R. A. 1990. Streamlining the design process: Running fewer subjects. In Proceedings of the Human Factors Society 34th Annual Meeting. ACM, New York, 291-294.
46. Virzi, R. A. 1992. Refining the test phase of usability evaluation: How many subjects is enough? Hum. Fact. 34, 457-468.
47. Woolrych, A. and Cockton, G. 2001. Why and when five test users aren't enough. In Proceedings of the Proceedings of IHM-HCI 2001 Conference, J. Vanderdonckt, A. Blandford, and A. Derycke Eds., Cépaduès Editions, London, UK, 105-108.